See the Server Plugin Development docs page which is intended for plugin developers to use to get started in writing new plugins.

This page is only for people that want to know how the internals of the server plugin container was designed/implemented and how to add code to create new types of plugin containers that run inside the master plugin container. Ignore this page if you just want to write your own server-side plugin. See the above tip for the link to the page you should read.

Server Side Plugin Container and Plugins

While not quite exactly like the agent-side plugin container system, we need a server-side plugin container as a singular means to deploy, update and keep in sync across HA servers, server-side plugins.

The server-side plugin facets may, in fact, have orthogonal functionality as compared to other facets (i.e. one facet may be completely isolated from the needs and requirements of other facets) but having a single plugin container will simplify the deployment model. We would like only one way to install server-side plugins; only one way to update them; only one way to keep all servers in-sync with which versions of plugins they have.

The plugin container will need to provide some common functionality that all plugins and their facets could use, such as:

A scheduling engine such that a plugin component method could be called periodically.
A configuration engine such that a user can configure each plugin with different settings, but while using the same configuration mechanism across all plugins (think "plugin configuration" for server-side plugins, just as it works for agent-side plugins).
A deployment model that works the same as the agent plugins. In fact we could just reuse the agent deployment model - we'd put server-side plugins in rhq-downloads/rhq-plugins right next to agent plugins. We would have to add a new column to RHQ_PLUGINS to flag a plugin as agent side or server side (a simple enum column would suffice). We would then immediately be able to reuse the plugin distribution mechanism that helps keep agent plugins the same across all RHQ Servers.

In RHQ 1.3, we have only a single type of server-side plugin - the content source plugins. There is one plugin container that hosts them. We need to expand that content source server side plugin container so it can host different types of plugins.

Server Side Plugin Types

Below are different types of plugins that we potentially would like to host on the server-side. This is not an exhaustive list; I'm sure there are many other types we can think of or need.

PackageSource

This is, essentially, the content source plugins we have today, just renamed as "Package Source" plugins since they are the source of package metadata and content from remote repos.

ChannelSource

This facet gets channel definitions from remote repos. Essentially useful for supporting remote repos such as RHN Hosted or the JBoss CSP. ChannelSource facet can also provide things such as child channels or channel groups.

SupportTicketIntegrator

This can integrate with external ticketing systems. The RHQ Support subsystem today has the ability to obtain snapshot reports which are zip files that contain things like log files, data files, configuration files and the like. This SupportTicketIntegrator could take that and upload the snapshot content and create support tickets. It could also be used to be scheduled such that it can check status of a resource and if a resource goes down or appears to act abnormally, a ticket could be created. As the resource's behavior changes, the ticket could be updated automatically ("resource had thread count >200 but recently that metric went down to <100").

EntitlementAccessChecker

This plugin facet can integrate with an entitlement service so it can check for the latest subscribed entitlements. It could perhaps spit out warnings if something is happening in the management environment that wasn't entitled. This checking would occur on some given schedule.

AlertHandler

This would provide a pluggable mechanism to the alert notification subsystem. When an alert is triggered, this plugin would be able to pass that alert to some other remote server (e.g. a Twitter message, or send the alert to HP ServiceDesk). This would be a "global" handler - it will be told about and handle all alerts that are triggered in the system, regardless of what alert definition or what resource triggered the alert.

We could attach the alert handler defined in the server side plugin to specific alerts. Thus allowing only certain alerts to be handled by these external services. These would be "alert-specific" handlers.

We could also have the alert handler run a CLI script on the server - this could perform server-side workflow functions, such as, "disable some alerts, then restart some resources, then execute some operations, then reconfigure some other resources".

PerspectivePlugin

This would register metadata for perspective UI integration. This is how we would be able to inject perspectives into the UI. This is an important plugin and probably one of the first we'd want to implement.

The perspective plugin would contain the perspective "descriptor" and the plugin container would have to know how to deploy that perspective properly.

Implementing the perspective deployment model this way would allow the perspectives to be deployed on all servers in the RHQ Server cloud.

IntegrationPlugin

Runs on user managed schedules with user managed configurations but does anything that the plugin wants. Essentially, this is just a way to run scheduled jobs but have the scheduled jobs have access to the server's SLSB API. (e.g. run morganator once a day to find new boxes)

QMFAdapter

This plugin could expose a QMF interface around the RHQ inventory. Thus making the RHQ server act as a QMF "resource" (or perhaps we expose each RHQ resource as a QMF resource? this is up in the air - just throwing some ideas out there).

WS-Management Adapter

Provides some remote services to provide WS-Management functionality.

ReportGenerator

Runs on given schedules to generate reports. This could be how we implement a JasperReports/Server integration.

Generic Plugin

This is a catch-all. The server-side plugin container will essentially just be a lifecycle-manager for all plugins. "initialize", "shutdown", "start" and "stop" will be the basic calls the PC will make to plugin components. Any generic plugin can put dumped in the PC and just have it be notified when it should initialize, shutdown, start and stop. The other plugin types described are essentially an extension to this "generic" plugin type - in fact, some of the ones above may just be generic plugins, simply because the PC may not interact with those plugins in any other additional ways other than these simple start and stop lifecycle methods.

RHQ Server Plugin To Manage Plugin Container

Today, the rhq-server plugin can manage the content source plugin container (it basically just interfaces with the plugin container's MBean). We want to continue this by allowing the RHQ Server plugin to manage this new content source plugin container. I don't know what metrics we'll want, or what operations or config, but we'll want this so at the very least developers and customer support can peer into the plugin container and see what its doing. We can take a look at the current plugin functionality and add to it.

Server-side Support For Agentless Management

There is a current set of ideas/designs surrounding the idea of "Agentless Management". It is unclear if we will need any server-side plugin support to provide agentless management capabilities, but it is mentioned here for completeness.

What Should We Do First?

We should probably refactor the current Content Source Plugin Container first. Get a generic plugin container that can have deployed within it any plugin. This provides a standard deployment and configuration mechanism for all future plugins. It will also provide a plugin update model that allows developers to (hopefully) hot-deploy plugins - which will only help them as they develop new plugins. Otherwise, developers will probably have to shutdown and restart the server after every new plugin update they perform.

We then refactor the current content source plugins to fit into this new plugin container model, thus verifying that the new plugin container "works". At this point, we could then begin to build out the new plugins as listed out above.

Note that the functionality within each specific plugin does not have to wait, however, getting them deployed and running in the server will need SOME deployment and configuration mechanism. So writing that each time for each specific plugin will be wasted effort since it would have to be fixed to fit into the new plugin container model. Howerver, things like alert subsystem APIs to inject new alert endpoints does not have to wait, since those APIs will be needed regardless of any deployment mechanism the alert plugins will need to integrate with.

Implementation

This section will talk about implementation details. As the implementation rolls out, this section will get updated.

Plugins source: http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/enterprise/server/plugins;hb=HEAD
Plugin Container source: http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=tree;f=modules/enterprise/server/jar/src/main/java/org/rhq/enterprise/server/plugin/pc;hb=HEAD

xml-schemas

Originally, all of RHQ's XML Schemas were in the core/client-api module. This even included the server-side-only content source schemas. This was wrong because the client-api module was meant to contain shared code between server and agent. These server-side plugins (their code and their schemas) are never to be referenced by the agent. Therefore, we need to move these into a more appropriate location. This is where the enterprise/server/xml-schemas module comes into play. We have moved the server-side content schemas and all classes related to the original content server-side plugin schemas to this new xml-schemas module. JAXB has been enabled in this module, so all schemas will get JAXB generated code for them and will be available in the xml-schemas jar file.

Sharing the Configuration schema
Games had to be played in the xml-schemas pom.xml in order for the server-side plugins to share the configuration schema (rhq-configuration.xsd) which currently resides in the core/client-api module. rhq-configuration.xsd must still be in core/client-api because the agent plugin container needs to parse configuration from agent plugin descriptors. But so will the server-side plugin container. To support this, the xml-schemas pom.xml will actually make a copy of the rhq-configuration.xsd into its target directory and will tell JAXB to use it when generating code for the other server-side plugin schemas. But to ensure we don't have duplicate copies of JAXB generated classes for the config schema, xml-schemas pom.xml will delete its own copy of the configuration JAXB classes and will instead rely on its dependency on client-api jar to get those classes.

There is a main rhq-serverplugins.xsd schema that defines the root element that all plugin types must support. Each plugin type will have its own schema that extends this core root schema (see rhq-serverplugin-generic.xsd for an example - you can normally copy this file and use it as a starting point for any new schemas you want to add). It must extend it using XML Schema's substitution groups. If you ever need to add a new plugin type, these are the things you need to do to add its new schema:

Copy rhq-serverplugin-generic.xsd and give it a name for your new plugin type, such as rhq-serverplugin-YOUR_PLUGIN_TYPE.xsd.
Rename the root element that will substitute the server-plugin root element, thus extending the schema to include your schema.
Add additional XML schema that your plugin will need so it can do its work. You can directly add XML schema here, or you can create your own separate .xsd for your plugin types custom schema and just import it so it can be placed inside the root element.
Add your schema information as new constants in org.rhq.enterprise.server.xmlschema.XmlSchemas
Add your new schema to the context path used by the ServerPluginDescriptorUtil - see the static block that adds map entries to org.rhq.enterprise.server.xmlschema.ServerPluginDescriptorUtil.PLUGIN_CONTEXT_PATH
Add a new unit test to make sure your new schema can be generated properly and used to parse your new plugin type's descriptor

Have the Master Server Plugin Container create your new plugin container by adding the appropriate line of code in the method: org.rhq.enterprise.server.plugin.pc.MasterServerPluginContainer.createPluginContainers()

Make sure you keep the XML Schema type, element and JAXB classes consistent with how the other schemas define them. This makes the schemas and the JAXB generated code easier to read and use since it will be consistent across all plugin types. e.g. root element type that will be substituted should be "<name>PluginDescriptorType".

org.rhq.enterprise.server.plugin.pc

The new plugin container code will be consolidated into the package org.rhq.enterprise.server.plugin.pc. The different plugin types will have handler code in subpackages here (e.g. org.rhq.enterprise.server.plugin.pc.content will house the content plugin code).

Plugins themselves are free to use whatever package layouts they want.

Dependencies

There will be no plugin-to-plugin dependencies. Unlike agent plugins, server-side plugins do not extend other plugins nor do they depend on other plugins. Each server-side plugin is independent from one another. There is no guarantee on the order in which server-side plugins are started or stopped.

Plugin Containers

There is a master plugin container whose job is to simply start and stop the individual plugin type plugin containers. So the MasterPluginContainer will start and stop the ContentPluginContainer, AlertPluginContainer, PerspectivePluginContainer, et. al. - one plugin container for each type of plugin we support on the server-side. Each of those plugin containers extend the AbstractTypePluginContainer class to pick up common functionality needed by all.

Each plugin type container will have the responsibility to deploy each plugin the way they need to be. The content plugin container will be able to schedule sync jobs when it starts up and will deploy content plugins appropriately. The alert plugin container will inject alert handlers into the alert subsystem as appropriate. Each plugin container will be able to do whatever they need to do in order to properly support their plugins.

A plugin can only be of one type. For example, you have a "content" plugin that defines repos and package sources, or an "alert" plugin that defines custom alert handlers.

This illustrates the "one master plugin container, N type plugin containers, M plugins":

                    MASTER PC
                     ___|___
                    /       \
                   /   ...   \
                  /           \
             CONTENT PC       ALERT PC
     ___________/               \__________________________
    /     /    /                 \           \             \
  RHN   CSP    YUM           IRC plugin  HPServiceDesk  Twitter plugin
plugin plugin plugin

Plugin Components

Any server-side plugin can define a plugin component to act as a lifecycle listener and scheduled job. This object is stateful and implements simple initialize, start, stop, shutdown methods that are invoked at the appropriate times. It is also invoked when scheduled jobs are triggered. For Generic plugins, this is the only mechanism by which you plugin into the core engine. The plugin component can do whatever it wants. You define this in the "plugin-component" element under the root XML element in the plugin descriptor. See the etc/samples/custom-serverplugin example.

Server Plugin Context

The plugin components will be given a plugin context that contains information about the plugin - its name, its descriptor XML, its configuration with more as we add things.

Plugin Configuration

Under the root XML element, you can add <plugin-configuration> which follows the standard RHQ configuration schema. This will let you define configuration for any plugin. The configuration is stored in the plugin context.

Disabling Plugins When Configuration Changes
> When editing a plugin configuration of a server side plugin,
> the plugin is disabled (which I think is wrong anyway).

https://bugzilla.redhat.com/show_bug.cgi?id=548526

I'll explain why this is.

First, if you have only a single server (i.e. you are not using multiple RHQ Servers in an HA Server Cloud), then yes this is not needed - we should just reload the plugin in the same state it was in (if it was disabled, leave disabled; if enabled, leave enabled). In fact, that could be an enhancement we should make - if there is only one server in the RHQ_SERVER table, then we should not disable the plugin.

But - think about the case when there is 2 or more RHQ Servers in the HA Server Cloud. Let's go through an example.

I have 2 RHQ Servers, server #1 and server #2. Each, of course, has their own set of server-side plugins running in their own master plugin container. As with everything else in our homogenous design, these 2 RHQ Server master plugin containers are independent of one another - they don't directly talk to each other, they don't know each other exists.

I point my browser to server #1 and through its GUI, I change a server plugin's configuration, persisting the change to the database. Server #1 knows the change was made - because I made the change through its GUI. So Server #1 knows enough to tell its master plugin container that it needs to reload the plugin whose configuration changed. BUT! How does server #2 know that this plugin's configuration changed? How does it need to know that it must get its master plugin container to reload the plugin? There is no way today for this to happen, short of the servers to periodically poll the database.

How do we solve this today? When a server changes a plugin configuration (like I did on server #1 in the above example), the server will disable the plugin. By default, all servers check into the database every 5 minutes to see if a plugin state changed (enabled->disabled, or disabled->enabled) and reloads the plugins as appropriate (see org.rhq.enterprise.server.core.plugin.ServerPluginScanner.registerServerPlugins()). So, in effect, this change to the disabled state IS the communication from one server to all others in the HA Server - its how one server lets the others know that they need to reload the plugin because its configuration changed.

This is not a good solution because it means to be absolutely sure all servers pick up a config change, the user must not re-enable a plugin for at least 5 minutes after a plugin config change was made. What we need to do is have each server periodically poll the plugin configuration as seen in the database and if a change has been detected, then it should reload its plugin (this would allow the state to remain as-is since the state value will not be used as the "change marker".

Scheduling

One of the common features the plugin container will provide plugins is scheduling. There is a scheduled-jobs element defined in the main serverplugin XML schema. It lets you schedule periodic invocations of the plugin component or other classes. The abstract pc (AbstractTypeServerPluginContainer) has a scheduleJob API that can be used by plugin containers to schedules jobs if they want their own way of doing scheduling, above what is provided.

Read more about the XML semantics in the XML Schema annotation documentation: http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterprise/server/xml-schemas/src/main/resources/rhq-serverplugin.xsd;hb=HEAD

Plugins "DropBox"

To make it easier on users and developers, a new plugin deployment mechanism has been added. There is now a "plugins" directory on the server at the top of the installation - it is a peer directory to /jbossas and /bin. This new "plugins" directory is called the "dropbox" because here you can copy any plugins (agent or server) and the server will detect new files in there and deploy them for you. The server will poll this directory periodically and when it finds a file, it will copy it to its appropriate location inside the server. Once the plugin is copied to where it needs to go, the server will delete it from your dropbox. If you ever want to update the plugin, just copy the new updated plugin jar to the dropbox, and it will get updated.

Since this dropbox is located at the very top of the installation, its easy to remember where it is and easy to get to it. Since it handles agent plugins and server plugins equally, you no longer have to think where a particular plugin goes based on the type of plugin it is. Just copy all plugins to /plugins dropbox and the server will take care of the rest.

Note that the plugin UI page still allows you to upload plugins, if you prefer to install plugins using a browser UI. The plugins dropbox will probably be used mostly by developers.

Design-Server Side Plugins