Contains the implementation of the external JMX APIs from org.apache.geode.admin.jmx.

## JMX Support in GemFire

Our goal was to provide JMX administrative support in GemFire. The design was influenced by these important factors:

1) Requirement to not impact performance of the product
2) Desire to introduce JMX without altering the existing product or the management console
3) Similar functionality already existed in the console and the internal.admin pkg which the console uses
4) Requirement to also introduce a simple and usable Admin API which or may not be related to JMX

From a functional stand point, the JMX support was supposed to provide most of the same administrative and operational monitoring that the console already provides. In some cases we limited the functionality due to security concerns and in others because it was hard to express the features as JMX beans. The JMX Agent also provides some functionality (such as Health monitoring) that is not currently in the console.

The Agent communicates with the distributed system using the same distribution manager {@link org.apache.geode.distributed.internal.DistributionManager} as the console uses and thus has the same requirements that determine what distributed system it can manage. Although the Console currently supports managing multiple distributed systems, we decided that a given Agent should only be able to manage a single system. We have not tested the use of more than one Agent for the same system, however nothing currently prohibits this.

We decided to develop a simple public Admin API which in essence wraps the internal.admin API that the Console currently uses extensively. The Admin API also contains implementations of new functionality not in internal.admin. Our JMX support is an extension to this Admin API. In an overly simplified view, the GemFire JMX MBeans are ModelMBeans that manage instances of the Admin API objects housed in the Agent's MBeanServer.

The selected architecture consists of a Daemon Agent, which exists in a separate VM that GemFire does not depend on. This Agent hosts an MBeanServer, instances of any and all MBeans registered for managing a GemFire distributed system, and server connectors/adaptors that various types of clients can connect to.

The two server connectors we selected are the HttpAdaptor and the RMI Connector. The HttpAdaptor provides an HTML user interface of all MBeans in the MBeanServer. Although this generic UI is not as rich as an RMI client (or the GemFire Console) could be, it provides a functional and easy to use UI with no development required. The JMX Remote specification details the standard connectors. Although the HttpAdaptor is not required by this Sun spec. it is included in some form with all JMX implementations that I investigated. It should be noted that our JMX Agent currently starts up the HttpAdaptor, but not the RMI Connector. The latter is deferred as later work since some form of client must be developed for testing. Further research may also uncover a generic, configurable open-source RMI client for JMX. The GemFire Console could in theory be reworked as an RMI Connector client, but this is not currently planned.

Two open-source JMX implementations made it to our final review for consideration: XMOJO and MX4J. The decision to go with MX4J was based mainly on our perceptions of MX4J being more active and widespread in use. Additionally, XMOJO is associated with AdventNet which produces commercial products. This made MX4J seem more true to open-source and safer from corporate tampering.

ModelMBeans are very dynamic and capable of managing aggregate resources. Use of a ModelMBean entails specifying meta-data to an instance of javax.management.modelmbean.RequiredModelMBean. This meta-data identifies the manageble resource(s) which can consist of a single object, or many objects, including those in one VM or in any number of distributed VMs. We decided to subclass classes in the Admin API in order to massage them a little and make them easier to use as a managed resource by the ModelMBean. For example, org.apache.geode.admin.GemFireManager represents a type of member in a GemFire system which manages shared memory. When an MBean is registered for managing the GemFireManager, the JMX Agent instantiates a "JMX friendlier" subclass: org.apache.geode.admin.jmx.internal.GemFireMangerJmxImpl. Comparison of this class with the non-JMX super class org.apache.geode.admin.internal.GemFireManagerImpl will illustrate what "JMX friendly" means better than trying to explain it here...

One standard approach to defining a ModelMBean is to programmatically build the necessary meta-data. The folks on the Tomcat approach developed a better solution... an XML definition file which Jakarta Commons-Modeler parses to create the meta-data objects required to definie the ModelMBean. We currently have our XML descriptor file at org.apache.geode.admin.jmx.mbeans-descriptors.xml. Commons-Modeler can be found at http://jakarta.apache.org/commons/modeler/

Here's a quick run-down of the Admin and JMX pkgs in GemFire...

org.apache.geode.admin

interfaces describing GemFire entities and resources for managing or monitoring

org.apache.geode.admin.internal

implementations of the admin pkg which could be used to managing GemFire
recommendation is to create one set on non-JMX unit tests for these and then wrap that unit test within another test that is specific to its use in JMX

org.apache.geode.admin.jmx

Commons-Modeler descriptor file mbeans-descriptors.xml
AgentMBean - non-required interface that simply represents a public documentation of what the Agent does
VersionMBean and Version - unused (but functional) examples of a standard MBean (other MBeans as defined in mbeans-descriptors.xml also provide version info)

org.apache.geode.admin.jmx.internal

subclasses of org.apache.geode.admin.internal classes that use JMX services, implement ManagedResource, register ModelMBeans to manage themselves, and other JMX related details
ManagedResource - simple interface we use to create a more uniform approach in using JMX
Agent - application with main, registers HttpAdaptor, acts as a factory to DistributedSystemJmxImpl (which is the entry point to using JMX with a GemFire system)
AgentConfig - configuration for Agent that reads/writes to a properties file
MBeanUtil - utility class with static methods for defining MBeans, registering for timer-based refresh notifications, and lots of other JMX stuff
StatisticAttributeInfo and ConfigAttributeInfo - these are the results of refactoring common code from some of the JmxImpl classes that required runtime dynamic (ie, not known at design time) MBean attributes
SpecManagedBean - subclass of part of Commons-Modeler, this class acts as a workaround for a bug in Commons-Modeler (I hope to remove this after working with Jakarta-Commons to correct the bug)
AgentPrintStream - this is a hack workaround for suppressing warnings from Xalan that derive from a purported incompatibility between the XSLT sheets in MX4J and the version of Xalan in JDK1.4.2 - experts with Xalan recommend upgrading to the very latest version of Xalan which is not currently desired for GemFire

### Some caveats, workarounds, and GemFire bugs to be aware of:

1) MX4J uses XSLT style sheets that are intended to work with the latest version of Xalan. JDK1.4.2 bundles an older version of Xalan which generates warnings.

2) MX4J's implementation of javax.management.modelmbean.RequiredModelMBean contains a bug such that any return type defined as an array of some object results in it attempting to Class.forName the class name with the "[]" still in the name of the class. Our current workaround is to return java.lang.Object where we'd like to return an array of some non-primitive type. I hope to look at this closer and work with MX4J to correct it (unless that latest code base at MX4J already has a fix).

3) Our MBeans currently have some return types of Admin interfaces and GemFire MBean types. This is not recommended. The correct approach is to return javax.management.ObjectName or an array of ObjectName so that remotability is not broken. We have a bug filed to correct this in GemFire.

4) Commons-Modeler provides a simple, incomplete implementation of ModelMBean called org.apache.commons.modeler.BaseModelMBean. We decided to use the standard RequiredModelMBean which all JMX implementations are required to supply. The JMX spec details several "managed resource types" that a ModelMBean can support. ObjectReference is the type that both MX4J's RequiredModelMBean and Modeler's BaseModelMBean support. However, MX4J is more strict in it's interpretation of the spec which spells it out as "ObjectReference". Modeler's BaseModelMBean performs no enforcement and simply assumes it is managing type ObjectReference. Modeler has a bug in org.apache.commons.modeler.ManagedBean because it specifies "objectReference" which is the incorrect case in comparison to the specification. I intend to work with the Jakarta-Commons folks to change this to comply with the spec. The Modeler will use BaseModelMBean by default even though it is actually depending on a JMX implementation such as MX4J or XMOJO. org.apache.geode.admin.jmx.internal.MBeanUtils tells Modeler to use RequiredModelMBean instead and also uses org.apache.geode.admin.jmx.internal.SpecManagedBean as a workaround to the whole "objectReference" issue. You could feasibly use org.apache.commons.modeler.BaseModelMBean or RequiredModelMBean.

### Capabilities currently supported in GemFire via JMX:

1) View overall system and its settings and start/stop all members
2) View all managers and applications
3) View and modify config (includes optional use of JMX Timer service to auto-refresh)
4) View any statistics (includes optional use of JMX Timer service to auto-refresh)
5) View member's cache and its statistics
6) View a cache's region and its attributes, statistics, but not the data
7) Health monitoring (ask David W. for more info... I haven't had a chance to look at this yet)
8) Create, start, and stop managers

### TODO:

1) Creation and control of Locators
2) Enable the RMI Connector and build tests
3) Bind Address support (bug 30898)
4) SSL Configuration support
5) JMX Connector security
6) more thorough automated tests!!
7) Documentation of use w/ built-in Connectors (RMI and HTTP)
8) Documentation of use w/ AdventNet SNMP Adaptor
9) Documentation of use w/ selected commercial management products
10) Determine high-availability requirements if any (mentioned by Jags)

See also deferred backlog from Hardening/Operations sprint

It's very easy to use the JMX Monitor service to monitor an MBean's attribute and fire JMX Notifications. Statistics and Config Parameters are "MBean attributes".

### Using Third Party Management Consoles/Tools

There is information on AdventNet's website detailing the use of their SNMP Adaptor to enable connectivity to HP OpenView, IBM Tivoli, CA Unicenter, and other SNMP managers.

Aside from using AdventNet's SNMP Adaptor, HPOpenView is the first I've seen of the big players offering what appears to be a way to plug our JMX support into their product for management of GemFire. I haven't had a chance to look at it yet. Most likely their SPI supports writing a small amount of glue code to have their console become an RMI client to our JMX VM (which is essentially an MBeanServer "server" that hosts our MBeans). I think it's unlikely that such vendors would want to support hosting of third party MBeans and the classpath/versioning headaches involved in that. The JMX Remote specification really is meant to cover how such console's would perform the remote connections to custom MBeans in external VMs/products. It is likely that the other management tool vendors such as Tivoli have a similar SPI available or in the works.

See HP Dev Resource Central's page on JMX for info and links on using JMX to integrate with HP OpenView. You can also sign up for HP Developer News

### Deferred Backlog from Hardening/Operations Sprint

Note: some of these tasks may no longer be worded properly or even needed...

Provide a consolidated view of all cache Regions via JMX
Send an alert upon system failure, communication failure, etc.
Monitor cache system failures, comm failures, loader failures, etc via JMX
Detect/handle stuck shared memory locks via JMX
Provide a "combination view" (distributed system-wide) of certain statistics via JMX
Integrate with Tivoli and produce a document
Test managing caches via JMX with a large number of Regions, Entries, etc.
Create XDoclet module to auto-generate mbeans-descriptors.xml (Note: we're using JavaDoc 1.4.2 which is incompatible with XDoclet; Sun intends to restore compatibility in the next version of JavaDoc)
Investigate AdventNet's JMX administration console
View percentage of VM memory used by a Cache via JMX
Investigate modifying the GemFire Console to use JMX
Investigate providing secure management via JMX
Document security usage for JMX management
Investigate managing GemFire via BMC Patrol
Investigate managing GemFire via HP OpenView
Investigate managing GemFire via UniCenter
Submit fix to Jakrta Commons-Modeler for ObjectReference fix in ManagedBean
Submit fix to MX4J for classload error in RequiredModelMBean
Enable the RMI Connector (currently commented out in Agent)
Create tests for RMI Connector
Document and JavaDoc usage of the RMI Connector
Add support to MBeanUtil to determine which MBeanServer the Agent is in
Correlate information from GemFire MBeans with MBeans from other products
Test deploying GemFire MBeans into other products' MBean containers