RHQ Frequently Asked Questions
General
What documentation is available for RHQ?
All RHQ documentation can be found at http://support.rhq-project.org/.
Is XXX database supported?
No. PostgreSQL and Oracle are the only supported databases, though there is interest in adding MySQL.
What is Java Service Wrapper and where does RHQ store its binaries on UNIX?
Java Service Wrapper is a third party utility RHQ uses to install the RHQ Server and RHQ Agent as a Microsoft Windows Service (so it can run as a boot time background service). Our usage of Java Service Wrapper is configured with the files rhq-server-wrapper.conf and rhq-agent-wrapper.conf. Java Service Wrapper is not used by RHQ on any UNIX platform - it is only used when RHQ is running on Microsoft Windows (and only when you elect to install the server or agent as a Windows Service). If you want to install the server and/or agent as a boot time background service on UNIX, you install their respective wrapper scripts in your UNIX platform's appropriate init.d location. See Running the Server and Running the Agent if you want to learn more about installing them as boot time background services.
How can I update the plugins on all my agents?
When you add a new plugin to your RHQ system, or you upgrade an existing one, you normally want to tell all of your agents to update their existing plugins with the new plugin versions. You can individually do this by executing the prompt command "plugins update" at any agent prompt. Or you can individually execute the operation "Update All Plugins" from the RHQ core UI's Operation Tab for each RHQ Agent. If you want to update all of your agents so they all download the latest plugins, you can use RHQ's DynaGroup feature along with the Group Operation feature to do this. First, create a DynaGroup (you do this from the Browse Resources page; click "Group Definitions" and then click the "New Definition" button). The DynaGroup definition should have the expression:
This creates a compatible group that dynamically adds all RHQ Agents as members to that group. Note that if you already have a compatible group with your agents as members, you can skip this group creation step.
Next, traverse to that compatible group that contains all your agents. You should see an Operations tab. From here, just invoke the "Update All Plugins" operation on that group. This will tell all of your agents in that group to update their plugins. Once that group operation is completed, all of your agent will have the latest, most up-to-date versions of all plugins.
Server
How do I get debug messages from the RHQ Server?
You can edit the jboss-log4j.xml configuration file to enable debug messages. Open jbossas/server/default/conf/jboss-log4j.xml in a text editor and change the "Threshold" for both the FILE and CONSOLE appenders to "DEBUG". You can optionally change other category priorities if you so choose.
Save the file and then restart the RHQ server (rhq-server.bat,sh). The debug messages can be found either in the console window or by examining the file logs/rhq-server-log4j.log.
In some cases, you will want to get debug messages from the RHQ Server launcher scripts. To do this, you need to set the environment variable RHQ_SERVER_DEBUG to any value. Now when you start via rhq-server.bat or rhq-server.sh, the launcher scripts will output debug messages.
 | Most log files emitted by the RHQ Server are found in logs. |
How can I specify command-line options for the RHQ Server JVM?
On UNIX
If you want to override the default max heap and permgen sizes, set them via the RHQ_SERVER_JAVA_OPTS environment variable, e.g.:
RHQ_SERVER_JAVA_OPTS="-Dapp.name=rhq-server -Xms256M -Xmx1024M -XX:PermSize=128M -XX:MaxPermSize=256M -Djava.net.preferIPv4Stack=true"
export RHQ_SERVER_JAVA_OPTS
Set all other JVM options via the RHQ_SERVER_ADDITIONAL_JAVA_OPTS environment variable, e.g.:
RHQ_SERVER_ADDITIONAL_JAVA_OPTS="-Dfoo=true"
export RHQ_SERVER_ADDITIONAL_JAVA_OPTS
On Windows
For all other JVM options, add "wrapper.java.additional.n" lines to <server-install-dir>\bin\wrapper\rhq-server-wrapper.inc, e.g.:
wrapper.java.additional.12=-verbosegc:file=gc-log.txt
wrapper.java.additional.13=-XX:+HeapDumpOnOutOfMemoryError
wrapper.java.additional.14=-XX:HeapDumpPath=heap-dump.txt
How can I debug JDBC access and trace SQL?
You can utilize a JDBC driver that logs calls to the JDBC API. There are some third party logger drivers, and RHQ ships with one in case you do not already have one (called LoggerDriver). Note that you should only do this when you are attempting to debug a problem that the RHQ Server is having with respect to SQL queries it is executing or its JDBC driver. Logging/tracing JDBC drivers is very inefficient. Here's how you enable your logger JDBC driver:
- Shut down the RHQ Server
- edit jbossas/server/default/deploy/rhq-ds.xml and change the connection URL as per the rules of your logger JDBC driver. If you are going to use RHQ's LoggerDriver, prefix the connection-url with jdbc:rhq-log: - keep the rest of the JDBC URL as-is and just add the string "jdbc:rhq-log:" to the beginning of it
- continue editing jbossas/server/default/deploy/rhq-ds.xml and change the driver-class to the name of your logger JDBC driver. If you are going to use RHQ's LoggerDriver, set driver-class to org.rhq.core.db.log.LoggerDriver
- If you are going to use RHQ's LoggerDriver, copy the following jars found in jbossas/server/default/deploy/rhq.ear/lib into jbossas/server/default/lib:
- rhq-core-dbutils-*.jar
- i18nlog*.jar
If you are using RHQ's LoggerDriver, the JDBC calls will be logged to stderr (and will be extremely verbose). If you only want to log SQL executions, set this system property before starting the RHQ Server:
jdbcLogSqlOnly=true
You can use the file logger which writes the debug logs to a file, as opposed to stderr. This requires you to set two system properties before starting the RHQ Server:
jdbcLogListener=org.rhq.core.db.log.FileLoggerListener
jdbcLogFile=<a full path to the log file; e.g. /tmp/rhq-jdbc.log>
To set additional system properties that the RHQ Server will pick up, you need to pass them in via -D Java options. Set the RHQ_SERVER_ADDITIONAL_JAVA_OPTS environment variable before starting the RHQ Server. If you are running on UNIX, you can do this, as an example:
export RHQ_SERVER_ADDITIONAL_JAVA_OPTS=-DjdbcLogSqlOnly=true
rhq-server.sh start
Another alternative to setting the environment variable to pass the -D options is to add those system properties directly in rhq-server.properties - they will be picked up as system properties by the RHQ Server at startup.
When do Baselines auto-calculate?
Go to the Administration page of the GUI and click the Server Configuration link. You will see a couple settings for Automatic Baseline Configuration Properties. Baseline Frequency determines how often the baselines will be calculated. By default it is 3 days. This means that every 3 days a new set of baselines are calculated (except for those that were manually set by the user - those remain pinned to the baselines set by the user). Baseline Dataset determines the minimum set of data that must have been collected for a measurement before a baseline for that measurement is calculated. The default is 7 days. So, for example, when it is determined that baselines should be calculated (every 3rd day by default), only those measurements that have data that are 7 days old or older will get a baseline calculated. Any measurements that do not yet have data from 7 days ago will be skipped. This ensures that when a measurement's baseline is calculated, you have a good representative set of data to include in the calculation (e.g. by default, you will have 7 days worth of data that will be included in the baseline calculation).
Where are the RHQ Server's Web Service WSDL documents?
The RHQ Server deploys its web services inside the JBossAS's web service container. This web service container provides you with a summary page that advertises all the web services/WSDL documents that it exposes. To see what web services are exposed, and to find out where their WSDL documents are, point your browser to http://localhost:7080/jbossws/services
I deleted a Platform from inventory. How do I get it be rediscovered, so I can re-import it?
RHQ 1.1 and later
Just force an Agent discovery by issuing the following command at the Agent command prompt:
RHQ 1.0.1 and earlier
On the machine corresponding to the platform you deleted, start the Agent if it is not already running:
<Agent-install-dir>/bin/rhq-agent.sh
Re-establish communication with the server by issuing the following Agent commands:
sending> shutdown
sending> start
You can now wait for the next scheduled discovery, or force an Agent discovery as follows:
The method above is preferred because the registered Agent is maintained. Alternatively, you can register a new Agent by restarting the agent, specifying the -l option, on the machine corresponding to the Platform you deleted. The Platform will get rediscovered. That is, first quit the agent (using the 'quit' command), then run:
<Agent-install-dir>/bin/rhq-agent.sh -l
How can I ignore an autodiscovered resource?
My agent discovered a new platform and found a few resources that I do not want to take into inventory. How can I do this?
First, you can just select the resources to import in the auto-discovery portlet and deselect the unwanted. As long as they are shown in the portlet, they are not imported. Of course this has the disadvantage that it might be confusing to always see them.
The other option is to select the resource you do not want to import and click on "Ignore", so it does no longer show up in the portlet. Although, if you try this on a resource on a freshly discovered platform, it will fail. The reason behind this is that the inventory is organized in a tree like manner with the platform as a tree-root and when a server or service is taken into the system (no matter if imported or ignored) it will be attached below that root. Now when the platform is not yet imported into the inventory, there is no root where the ignored resource can be attached to.
So to ignore a server on a platform: first import the platform and leave that server unchecked. When the platform is successfully imported, select the server and click on "ignore".
From the above explanation you can see that it is not possible to ignore just a platform. If you want to ignore a platform, just do not run an agent on it.
How does RHQ integrate with external LDAP user repositories?
RHQ uses passwords to authenticate users. Authentication information, comprising user names and passwords, can be stored in an internal database (the default) or in an external LDAP repository. It is important to note that RHQ support for LDAP currently does not include storing attributes other than user names and passwords. In particular, authorization information such as roles used to control access to RHQ resources is persisted in the internal database.
- Configuring RHQ to use LDAP for authentication
- In order to configure RHQ to use LDAP for authentication, navigate to the Server Configuration page (Dashboard->Administration->Server Configuration). The following configuration parameters can be specified:
- URL of the LDAP server: This defaults to ldap://localhost on port 389 (or port 636 if the SSL option is selected).
- Username/Password: The username and password to connect to the LDAP server.
- Search Base: Base of the directory tree to search for usernames and passwords while authenticating users, e.g., o=JBoss, c=US
- Search Filter: Any additional filters to apply when doing the LDAP search. This is useful if the population to authenticate can be identified via given LDAP property, e.g., RHQUser=true
- Login Property: The LDAP property that contains the user name. Defaults to cn. If multiple matches are found, the first entry found is used.
- Use SSL: Provides the option to use SSL while communicating with the LDAP server.
- The configuration settings are captured and stored in the internal database. No attempt is made to validate the information at this point: any misconfiguration would be detected when a user attempts to log in to the RHQ console.
- Authenticating users via LDAP
- Once the RHQ server has been configured to use LDAP for authentication, subsequent attempts to login to the RHQ console result in requests to the LDAP server to validate users' credentials. Communication with the LDAP server is handled by a class that implements a JAAS Login Module for LDAP. The login module first searches the set of base directories for a matching username applying any search filters. If a matching name is found, a bind request specifying both the username and password is sent to LDAP to validate the credentials. Authentication is deemed successful if the bind request returns normally.
- Irrespective of whether LDAP is selected for authentication, the credentials of the root user rhqadmin are stored in the internal database. Stacking login modules makes for seamless authentication: when the LDAP option is selected, normal users are authenticated in LDAP and the root user rhqadmin is authenticated in the database.
- Impact of user administration on the LDAP repository
- As stated earlier, RHQ uses LDAP only to perform credential validation. Auxiliary information about a user such as first/last name, phone number, email address, and RHQ roles is stored in the RHQ internal database. Furthermore, user administration actions performed in RHQ do not impact the LDAP repository. For instance, the LDAP repository is not populated with the username and password when a user is registered in RHQ. The user must be defined in the LDAP repository independently of RHQ administration, and is assumed to have credentials populated in the LDAP repository when he/she is ready to access the RHQ console. In other words, RHQ uses the LDAP repository in read-only mode.
- One of the interesting features of LDAP integration in the product is the support for self-registration in RHQ available to those who are identified as potential RHQ users in the LDAP repository. One way of identifying RHQ users in the LDAP repository is to define attributes that can be specified in a search filter in the RHQ configuration, e.g., RHQUser=true. When such a user accesses the RHQ console for the first time, he/she is first authenticated in the LDAP repository, and then redirected to the RHQ registration page to capture auxiliary information such as first/last name and email address. This alleviates the task of user registration for RHQ administrators, and reduces the likelihood of errors as information is entered directly by the registrants.
RHQ does currently neither check server certificates for ldap over ssl, nor can it provide client side certificates to the LDAP server.
Agent
How do I get debug messages from the RHQ Agent?
Before starting your RHQ Agent, set the environment variable RHQ_AGENT_DEBUG to any value. Now when you start the agent, both the launcher scripts and the agent itself will output debug messages. The logs can be found in the logs directory. If you are launching the RHQ Agent on Microsoft Windows using the service wrapper, you must set RHQ_AGENT_DEBUG and then install the service via rhq-agent-wrapper.bat install. You can alter what the agent logs by modifying the config/log4j.xml file (or config/log4j-debug.xml if RHQ_AGENT_DEBUG is set). Note that modifying these files requires an agent restart in order to pick up the changes.
How do I start the RHQ Agent fresh, as if newly installed?
If you want the agent to clean itself of all previous inventory and force itself to re-register with the server, shutdown the agent and restart it with the --cleanconfig command line option. If you do this, you also can pass in the --config argument to have it start up with a configuration file you specify (otherwise, the default agent-configuration.xml will be used). The -l option is an alias for --cleanconfig and -c is an alias of --config - so your command line can look like this (if on Windows, replace .sh with .bat):
or
where both clean the old configuration, but the first loads the default agent-configuration.xml and the second loads your custom my-agent-configuration.xml (it will look in the conf/ directory, unless you specify a full path to a location other than conf/).
I want to run agents on all my machines, but only one starts OK - the rest fail due to binding to a wrong address
If you want to run multiple agents, but many fail to start with this error:
then there are a couple things you need to consider.
First, if you changed your agent-configuration.xml manually (say, to change IP addresses), did you do that after you initially setup the agent? The agent's configuration XML file is not referenced after the agent is setup - it doesn't need to because its configuration is persisted using Java Preferences (this is so it can support agent updates or agent re-installs without losing its configuration). If you want to change the agent's configuration file and have those changes picked up, restart the agent and pass it the --config command line option (or -c which is shorthand for --config). This tells the agent to re-read the configuration file and make that its configuration, overriding any old configuration it persisted before.
The other question to ask is - is your home directory stored on NFS? If so, then you are probably picking up the same Java Preferences across all your machines (see $HOME/.java - that is the default location where Java stores Java Preferences on UNIX - on Microsoft Windows, it goes in the registry so this might not be relevant if you are on Windows). If you are running the agents as the same user and your user's home directory is shared (via NFS or some other sharing technology) then one solution is to have your agents use different Java Preferences names. To do this, edit your agents' agent-configuration.xml files and change their Java preferences node names from "default" to something else that makes them unique across all agents. For example, change:
to
Now each time that you start your agents, you need to tell them where they can find their preferences (since you overrode its default location). You tell the agent your new preference name via --pref (or its shorthand notation of -p). Since you changed the configuration file, don't forget that the first time you restart the agent you need to pass in -c too):
Thereafter, the agent need only be passed in the -p every time you restart it (the new configuration will be persisted for you).
Read the comments at the top of agent-configuration.xml, it has some relevant info in there. You can also read the usage help too: rhq-agent.sh --help.
If you do not want to be forced to edit your configuration files and pass the -p option, the other alternative is that you can define the system property java.util.prefs.userRoot to point to some other, unique, location (e.g. /etc/rhq-agent-prefs). When the agent starts, Java will use the value of that system property as the location where it will store its Java Preferences. You set this system property on the agent via the environment variable RHQ_AGENT_ADDITIONAL_JAVA_OPTS. When you set that environment variable, rhq-agent.sh will add its value to the default set of Java options when passing in options to the agent's Java VM:
My agent just hangs on startup
The agent stores its configuration in the java preferences system. When you start it for the very first time, the preferences are empty and the agent starts to query you for some parameters. On the next start it will open the preferences, see that they are not empty and take the parameters from there.
If you now start a completely freshly unzipped agent, it will not have a token from the server yet, but think it already was fully configured from the data in preferences. In that case it will hang with "The agent does not have plugins - it will now wait for them to be downloaded..."
On the server side you should see something along
If this is the case, you need to start the agent with the -l switch as described above.
For more information also have a look at some detailed explanation below.
Log messages
What are "Command failed to be authenticated" messages?
Agents are assigned security tokens when they first register with the server. The token is one way an agent identifies itself with the server. If an agent does not identify itself with any token, or if it identifies itself with a wrong token, the server will deny access to that agent - in other words, the server will reject commands that come from that agent until that agent has properly registered. If an agent is consistently causing "failed to be authenticated" errors on the server, then it usually means the agent has been misconfigured, or it is an unknown agent attempting to identify itself as another agent (this will happen if you rebuilt the RHQ Server with -Ddbsetup, meaning your database has been cleaned and all agents have been wiped from the RHQ Server database). Restart your agent with the "-l" command line option to clean out its configuration and allow it to be accepted by the RHQ Server.
A common failure message is the following:
This one can typically be ignored. Notice the command type [identify] - this is a message the agent sent to the server immediately upon startup. All agents immediately attempt to identify the RHQ Server when they startup as part of their RHQ Server Auto-Discovery/Polling feature. However, the agent may not yet know its security token, and hence will attempt to identify the server as an unauthenticated agent. In this instance, the server will reject the request. Not to worry - the agent attempts to identify its server periodically, so even if this one request failed, it will attempt to identify the server later. Eventually (usually quite quickly), the agent will obtain its token and will successfully be able to identify the server. So, in short, if you see just a few "failed to be authenticated" commands of type "identify" in your server log files, they can safely be ignored. PLEASE NOTE Do not rely on the security token mechanism as a way to protect your RHQ environment from intrusion. If you require secure communications between servers and agents, see the Securing Communications section to learn how to setup SSL for authentication and encryption.
What are "fail-safe cleanup" messages?
You'll often see messages in your logs that look like:
Ignore them. They are perfectly normal and expected. They deal with the underlying ORM technology used (Hibernate) and how it automatically cleans up after itself to prevent memory leaks.
PostgreSQL Plugin
Why is the agent showing an error in my PostgreSQL discovery about authentication failed for user "postgres"?
The PostgreSQL plugin attempts to log into the database server using the username and password of "postgres". In many installations, this is a default superuser and will work. However, it is also possible that this login could fail for a number of reasons:
- The "postgres" user has been deleted.
- The password for the "postgres" user has been changed.
- On Linux, the administrative login has been set to "ident sameuser".
In many cases, this can be alleviated as follows:
- Inventory the discovered Postgres resource. Its availability will show as down and it will not find any child resources.
- Navigate to the inventory tab for the Postgres resource.
- Under Connection Properties, click the Edit button.
- Change the "role name" and "role password" fields to reflect a valid super user account on the Postgres instance.
Additionally, Postgres may need to be changed on Linux systems to allow password based logins (i.e. "md5" v. "ident sameuser" settings in the pg_hba.conf file). Consult the Postgres for more details.
Why are most of the metrics for my Postgres resource showing up as NaN?
In many installations, Postgres will not start its statistics collector by default. To enable statistics collection, add (or change) the following line in the postgres.conf file:
stats_start_collector = on
How many database connections are necessary to monitor a Postgres database?
Each Postgres database inventoried in RHQ requires 1 connection.
Why can't I drop my database that is inventoried in RHQ?
With the frequency of availability and statistics monitoring, the Postgres plugin keeps an open connection to the database. As such, when attempting to drop a database currently inventoried in RHQ, an error will be thrown about the database being in use. In order to drop the database, the RHQ Agent monitoring the database must be shutdown or the database resource should be removed from RHQ. This will close the Postgres plugin's connection to the Postgres server and thus allow you to drop the database.
Apache Plugin
Where can I get the connectors?
The Apache plugin monitors an Apache Web Server via custom modules like the SNMP connector. You can download the open-source versions of these connectors and install them in your Apache Web Server.
Misc
The RSS Feeds on the dashboard don't work
This seems to be an issue with the Safari browser. Try to use a different browser like Firefox or a dedicated rss feed reader.
The execution of a Script-resource fails on Unix
When I invoke the "Execute" operation on a Script resource, it immediately fails and I get an error saying that the script can not be executed.
Make sure that the execute bit is set on the resource. You can set it via chmod +x scriptname
Installer fails on PostgreSQL with "Relation RHQ_Principal does not exist"
First make sure that the RHQ server / installer is allowed to connect to PostgreSQL. You might want to have a look at pg_hba.conf for this. If this is ok, and the installer is even able to create the RHQ database, but still fails with that error when you click "Install Server", then jut create an empty table with the name "rhq_principal" and retry the install. Tell the installer that you want to overwrite the existing schema.
