Performance Testing

RHQ Project

Current Environment (July 31st)

Machine AgentVMs by SpawnedAgents Standalone Agent? Server? perftest.agent.vm.sysprops.default Start Port/Incr
jon06.DOMAIN.COM
jon07.DOMAIN.COM 10 by 10 -Don.perftest.scenario=configurable-5
-Don.perftest.server-d-count=3
-Don.perftest.service-d-metrics-count=120
-Drhq.trace-command-config=true		 27000/50
jon08.DOMAIN.COM 10 by 10 -Don.perftest.scenario=configurable-5
-Don.perftest.server-d-count=3
-Don.perftest.service-d-metrics-count=120
-Drhq.trace-command-config=true		 28000/50
jon09.DOMAIN.COM 10 by 10 -Don.perftest.scenario=configurable-5
-Don.perftest.server-d-count=3
-Don.perftest.service-d-metrics-count=120
-Drhq.trace-command-config=true		 29000/50
jon10.DOMAIN.COM

All perf related .sh files referred to below can be found at http://jon01.DOMAIN.COM:8042/perf/ (/home/TEST_USER/httpd/htdocs/perf) and attached to this page.

Prepare Server and Agent build

Process: Need to do an initial check and build, then need to do an update(optionally clean) build and upload

  1. Do this on jon01, since that's the nfs server you'll likely to have fewer file system problems. Also it has Apache running already which we will utilize to distribute the files we'll generate
  2. Ensure you have Java, SVN and Maven set up in your environment. If necessary, add this to your environment: 
    export JAVA_HOME=/qa/tools/opt/jdk1.5.0_15
export MAVEN_HOME=/qa/tools/opt/maven-2.0.9
export SUBVERSION_HOME=/qa/tools/opt/subversion-1.4.0
export PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$SUBVERSION_HOME/bin:$PATH
export LD_LIBRARY_PATH=$SUBVERSION_HOME/lib
export LANG=en_US.iso88591
  3. Make room for the src 
    mkdir -p ~/perf/src
cd ~/perf/src
  4. Check out the code. If this is the first time: 
    svn co http://svn.rhq-project.org/repos/rhq/trunk/ rhq/trunk
svn co http://anonsvn.jboss.org/repos/jopr/trunk/ jopr/trunk

    Otherwise, you'll typically want to do something like the following, replacing the revision numbers with ones taken from a working TeamCity build.

    svn update -r 3355 rhq/trunk
svn update -r 239 jopr/trunk
  5. Copy the settings.xml attached to this page to here: ~/.m2/settings.xml
    1. Update rhq.rootDir appropriately to point at rhq/trunk
  6. Build RHQ
    1. Need to specify the ojdbc-driver profile, and jon01 repo in settings.xml, to get the oracle jdbc driver. 
      cd ~/perf/src/rhq/trunk
mvn -Pdev,enterprise clean
mvn -Pdev,enterprise,jon05-oracle,ojdbc-driver install
    2. If you get 
      [ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] An Ant BuildException has occured: The following error occurred while executing this line:
/home/ccrouch/perf/src/rhq/trunk/modules/core/dbutils/src/main/scripts/dbsetup-build.xml:321: Database schema is not up-to-date. Use -Ddbsetup

      then you're database is out of date you'll eventually need to run mvn -Ddbsetup-upgrade -Pjon05-oracle install, but for right now you can do the following to get RHQ built:

      mvn -Pdev,enterprise clean
mvn -Pdev,enterprise,jon05-oracle,ojdbc-driver -Ddbsetup-do-not-check-schema install
  7. Build Jopr. If this is not the first time you are building you should probably add the 'clean' target before 'install'. 
    cd ~/perf/src/jopr/trunk
mvn -Pdev clean install
  8. Distribute Server and Agent 
    cd ~/perf/src
# Download the necessary bits
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-server.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-agent.sh
chmod +x *.sh
./bundle-server.sh
./bundle-agent.sh
# The Server (dev-container.tar.gz) and Agent (agent.zip) should be available at http://jon01.DOMAIN.COM:8042/perf/

If there were database changes that need to be applied

  • Make sure to turn off any servers connected to the db before progressing
  • cd ~/perf/src/rhq/trunk/modules/core/dbutils
  • mvn -Pjon05-oracle -Ddbsetup-upgrade install
  • or the following if you want to blow everything away and start from scratch
    • # mvn -Pjon05-oracle -Ddbsetup install

Prepare AgentSpawn build

cd ~/perf/src/rhq/trunk/etc/agentspawn
# build the AgentSpawn src
mvn clean install
cd ~/perf/src
# Download the necessary bits
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-spawn.sh
chmod +x *.sh
./bundle-spawn.sh
# The AgentSpawn distro (agentspawn.tar.gz) should be available at http://jon01.DOMAIN.COM:8042/perf/

Using Server build

Process on each box: Need to do an initial run, then deal with having an existing server in place and running a new one

  1. First time through make sure you have
    1. Make sure sshd doesnt timeout and kill your session too quickly
      1. Edit /etc/ssh/sshd_config and update the properties as follows
      2. ClientAliveInterval 300
      3. Then restart sshd: /sbin/service sshd reload
      4. Logout and in again
    2. Java installed (in the PATH and JAVA_HOME set via .bash_profile)
    3. any firewalls turned off (/etc/init.d/iptables stop)
    4. date/time on the box is accurate (need root for this)
      1. if date is more than 1000s (15mins) away from current time then set it approximately e.g. : date -s "Wed Jul 15 10:21:46 EDT 2009"
      2. set date accurately: ntpd -qd
      3. start ntpd server: ntpd
      4. make sure ntpd starts on reboot: chkconfig --level 35 ntpd on
    5. then do 
      mkdir -p `hostname -s`/perf
  1. Each time through 
    cd ~/`hostname -s`/perf
# Download the necessary bits
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-server.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-server.sh
chmod +x *.sh
# setup the server
./unbundle-server.sh
./tweak-server.sh
# startup the server
./server/trunk/dev-container/bin/rhq-server.sh start

Old way to upgrade (the new unbundle-server.sh will do the shutdown/backup for you now)

cd ~/`hostname -s`/perf
cd ./server/trunk/dev-container/bin
./rhq-server.sh stop
sleep 30
cd ~/`hostname -s`/perf
# you may need to do the next step on jon01 if you get NFS errors
rm -rf ./server_old
mv server server_old
# Repeat steps above

Using Agent build

This is a prerequisite for using the AgentSpawn build

  1. First time through make sure
    1. All the server specific first time steps above have been done
    2. Ant 1.6.5 installed (in the PATH and ANT_HOME set via .bash_profile)
    3. then 
      mkdir -p `hostname -s`/perf
  1. Each time through 
    cd ~/`hostname -s`/perf
# Download the necessary bits
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-agent.sh
chmod +x *.sh
./unbundle-agent.sh

Using AgentSpawn build

  1. First time through make sure the prerequisites for using the Agent above have been done
  2. Each time through
    1. Get the bits 
      cd ~/`hostname -s`/perf
# Download the necessary bits
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-spawn.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-spawn.sh
chmod +x *.sh
./unbundle-spawn.sh
    2. setup for the appropriate agent 
      #jon07
./tweak-spawn.sh 10 27000 50 10 "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
#jon08
./tweak-spawn.sh 10 28000 50 10 "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
#jon09
./tweak-spawn.sh 10 29000 50 10 "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
    3. start 'em up 
      cd ./spawn/trunk/etc/agentspawn/src/scripts
# distribute the spawned agents
ant copy
# start them up
ant start

To upgrade

cd ~/`hostname -s`/perf
cd ./spawn/trunk/etc/agentspawn/src/scripts
ant kill
# get rid of any existing ones
ant clean
# Optionally delete the agent and get a new version of that
# Repeat steps above

See Also

Quick Re-Start

Here's a quick set of instructions that should work to get you going in the minimum number of copy-n-paste steps. You must have already performed all the build initialization steps mentioned above at least once. But once you have your build environment setup and you are building the perf environment, you can just copy-n-paste the steps below to do the builds quicker.

Build Server and Agent - do this on your BUILD machine

Build Server and Agent
# BUILD SERVER/AGENT
cd ~/perf/src/rhq/trunk
svn up
cd ~/perf/src/jopr/trunk
svn up
cd ~/perf/src/rhq/trunk
mvn -Pdev,enterprise,jon05-oracle,ojdbc-driver clean install
cd ~/perf/src/jopr/trunk
mvn -Pdev clean install
cd ~/perf/src
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-server.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-agent.sh
chmod +x *.sh
./bundle-server.sh
./bundle-agent.sh
# BUILD AGENT SPAWN
cd ~/perf/src/rhq/trunk/etc/agentspawn
mvn clean install
cd ~/perf/src
wget -N http://jon01.DOMAIN.COM:8042/perf/bundle-spawn.sh
chmod +x *.sh
./bundle-spawn.sh

Start New Server - do this on each SERVER machine

Restart Server
# START NEW SERVER
cd ~/`hostname -s`/perf
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-server.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-server.sh
chmod +x *.sh
./unbundle-server.sh
./tweak-server.sh
./server/trunk/dev-container/bin/rhq-server.sh start

Restart Agents - do this on each AGENT machine

Restart Agents
# RUN AGENT SPAWN
cd ~/`hostname -s`/perf
cd ./spawn/trunk/etc/agentspawn/src/scripts
ant stop
sleep 10
ant kill
ant clean
cd ~/`hostname -s`/perf
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-agent.sh
chmod +x *.sh
./unbundle-agent.sh
cd ~/`hostname -s`/perf
wget -N http://jon01.DOMAIN.COM:8042/perf/unbundle-spawn.sh
wget -N http://jon01.DOMAIN.COM:8042/perf/tweak-spawn.sh
chmod +x *.sh
./unbundle-spawn.sh
# !!!!!!!!!!!!!!! START: THIS IS DIFFERENT FOR EACH AGENT BOX !!!!!!!!!!!!!!!
./tweak-spawn.sh.CHANGEME xx xxxxx xx xx "-Don.perftest.scenario=configurable-5 -Don.perftest.server-d-count=3 -Don.perftest.service-d-metrics-count=120 -Drhq.trace-command-config=true -Drhq.agent.plugins.configuration-discovery.period-secs=0" "--nonative"
# !!!!!!!!!!!!!!! END: THIS IS DIFFERENT FOR EACH AGENT BOX !!!!!!!!!!!!!!!
cd ./spawn/trunk/etc/agentspawn/src/scripts
ant copy
ant start
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.