Distributed RHQ Environment Provisioner

Functionality

This tool would do the following:

(optional - probably a v2 feature) download a PostgreSQL distribution from the specified URL and use SSH to push it to the specified machine and install it there
download an RHQ distribution from a specified URL (could be a local file, a final release from SourceForge, or a nightly build from Hudson)
via SSH, push the distribution to the specified machines and install RHQ Servers on those machines using our automated/silent Server install mechanism; the rhq-server.properties used would be provided to the provisioning tool and template vars would be realized for
Server-specific props such as the Server IP
start the Servers in headless mode via SSH
download the Agent distribution from one of the Server's download page
install n Agent instances to the specified machines by either a) invoking our agentcopy tool via SSH or b) calling our RemoteInstallManagerRemote SLSB
start the Agents in headless mode via SSH
wait for Agent autodiscovery to complete then use CLI script to import all discovered Resources
(optional) create a bunch of compatible and mixed groups containing the inventoried Resources
(optional) create a bunch of test roles and users, with varying levels of authorization
execute SQL and/or CLI script to populate the DB with fake past data for all
- metrics - 7 days worth of raw data for all enabled metrics - use Greg's DataInserter class as a starting point
- avail - n days worth of ups and downs
- calltimes - n days worth of calls
- events - n days worth of events
- (optional/v2) - plugin config updates, resource config updates, operation executions

The tool would initially have a command-line interface but a Web interface could be added later.

Uses

The tool could be used for provisioning environments for a variety of purposes:

large-scale perf testing
demos
day-to-day dev testing

Example

To configure an HA perf environment with 2 Servers and 2 Agent machines running 50 Agents each, the tool would be invoked as follows:

java -jar rhq-env-provisioner.jar rhq-perf-env.properties

And rhq-perf-env.properties would look something like:

### RHQ Server Provisioning Specs
serverDistributionZipUrl = https://hudson.example.org/hudson/job/rhq-master/lastSuccessfulBuild/artifact/modules/enterprise/server/container/target/rhq-server-4.1.0-SNAPSHOT.zip

server[0].host = perfserver1.example.org
server[0].sshUser = rhq
server[0].sshKeyPassword = rhq
server[0].serverInstallParentDir = /home/rhq/perf-env
server[0].env.RHQ_SERVER_JAVA_HOME = /usr/java/jdk1.6.0_25
server[0].env.RHQ_SERVER_MAX_HEAP_SIZE = 2000M
server[0].env.RHQ_SERVER_MAX_PERM_SIZE = 400M

server[1].host = perfserver2.example.org
server[1].sshUser = rhq
server[1].sshKeyPassword = rhq
server[1].serverInstallParentDir = /home/rhq/perf-env
server[1].env.RHQ_SERVER_JAVA_HOME = /usr/java/jdk1.6.0_25
server[1].env.RHQ_SERVER_MAX_HEAP_SIZE = 2000M
server[1].env.RHQ_SERVER_MAX_PERM_SIZE = 400M

### RHQ Agent Provisioning Specs
agentMachine[0].host = perfagent1.example.org
agentMachine[0].sshUser = rhq
agentMachine[0].sshKeyPassword = rhq
agentMachine[0].agentInstances = 50
agentMachine[0].agentPortRangeStart = 27000
agentMachine[0].agentPortRangeIncrement = 50
agentMachine[0].agentInstallParentDir = /home/rhq/perf-env
agentMachine[0].agentNamePrefix = perfagent1-
agentMachine[0].agentEnv.RHQ_AGENT_JAVA_HOME = /usr/java/jdk1.6.0_25
agentMachine[0].agentEnv.RHQ_AGENT_JAVA_OPTS = -Xms150M -Xmx150M -Djava.net.preferIPv4Stack=true 
agentMachine[0].agentEnv.RHQ_AGENT_ADDITIONAL_JAVA_OPTS = -Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true

agentMachine[1].host = perfagent2.example.org
agentMachine[1].sshUser = rhq
agentMachine[1].sshKeyPassword = rhq
agentMachine[1].agentInstances = 50
agentMachine[1].agentPortRangeStart = 27000
agentMachine[1].agentPortRangeIncrement = 50
agentMachine[1].agentInstallParentDir = /home/rhq/perf-env
agentMachine[1].agentNamePrefix = perfagent2-
agentMachine[1].agentEnv.RHQ_AGENT_JAVA_HOME = /usr/java/jdk1.6.0_25
agentMachine[1].agentEnv.RHQ_AGENT_JAVA_OPTS = -Xms150M -Xmx150M -Djava.net.preferIPv4Stack=true 
agentMachine[1].agentEnv.RHQ_AGENT_ADDITIONAL_JAVA_OPTS = -Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true

### RHQ Server Configuration Properties (will get copied into each Server's rhq-server.properties file)

# Database
rhq.server.database.connection-url=jdbc:postgresql://perfdb.example.org:5432/rhq
rhq.server.database.driver-class=org.postgresql.Driver
rhq.server.database.xa-datasource-class=org.postgresql.xa.PGXADataSource
rhq.server.database.user-name=rhqadmin
rhq.server.database.password=1eeb2f255e832171df8592078de921bc
rhq.server.database.type-mapping=PostgreSQL
rhq.server.database.server-name=127.0.0.1
rhq.server.database.port=5432
rhq.server.database.db-name=rhq
hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect

# Quartz
rhq.server.quartz.driverDelegateClass=org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
rhq.server.quartz.selectWithLockSQL=SELECT * FROM {0}LOCKS ROWLOCK WHERE LOCK_NAME = ? FOR UPDATE
rhq.server.quartz.lockHandlerClass=org.quartz.impl.jdbcjobstore.StdRowLockSemaphore

Implementation Details

the remote agent install code in server-jar makes SSH calls using the JSch lib (see SSHInstallUtility) - this can be copied/reused
package the utility as an executable jarjar or uberjar, so it will be neatly contained in a single jarfile, and no shell scripts, batch files, or maven assembly scripts will need to be written

TODO

look at the agentcopy.properties file to get a feel for how it does the configuration of all the agents. that might spark some additional ideas

Design - Distributed RHQ Environment Provisioner