J

Product Overview

J-FORCE® is an acronym for JEDIS - Framework Oriented Remote Communication Environment. The J-FORCE® product provides a ‘framework’ for a system of distributed interacting processes, such as PASCOL®.

The J-FORCE® framework is designed as a code platform for systems having the following characteristics:

Client-server
Distributed
Multiple communicating processes; multi-threaded inclusive.
Multiple system instances
Controlled automatic deployment of configuration data between instances
Process monitoring and automatic recovery.
True 7x24 High Availability leverage with third party Hardware Vendor products.

J-FORCE® is built using the Common Object Request Broker Architecture (CORBA) standard with an Object Request Broker (ORB) abstraction such that a variety of standard compliant ORB products can be implemented. By default J-FORCE is shipped and licensed with a default ORB - OmniORB™ from ATT Research.

J-FORCE® provides an C++ development framework for applications to be developed leveraging real-time High-Availability without the nuances of rigid application design for implementation or business application limitations.

J-FORCE® application components are built with database independence offering an Oracle, Sybase or Microsoft SQL-Server abstraction.

J-FORCE® offers an Object Orientated approach to a distributed component and installation implementation.

J-FORCE® introduces a configuration object and manager thereof enabling "C++ Class object data" to be defined abstract to database type (s); stored externally or encapsulated as a image, byte or text types ; serially referenced with portable streams for external come inter-operable access.

J-FORCE® uses J-Tools® for common class and/or API reference.

Contents

Concepts

Framework Architecture

Host Manager

Domain Manager

Configuration Manager

Deployment Manager

Authenticator

Communication

Queuing

J-Tools

Application Components

Queues

Sub-Systems

Abstraction Layers

Databases

ORB's.

Thread Policies

Resource Management

Component Classes

CORBA / IDL

High Availability

MC Service Guard

SUN Cluster

Operating Systems/Development Kits

HP/UX

Sun Solaris

Microsoft Windows

Concepts

A system using the J-FORCE® framework has two main parts: the framework itself, which is common to all systems, and the application which will be different for each system. For most applications there will exist several server programs . Each of those servers may themselves consist of multiple co-operating component processes; this aspect is handled automatically by the J-FORCE® framework.

Some of the key J-FORCE® concepts are noted below :

Domain	A domain is a distinct partitioned-off area of operations. Domains may correspond to stages in product development, or to business areas, or any other way of dividing up the system or the activities of a business. It is hard to define comprehensibly without resorting to some examples: Production Testing Development A domain is a logical concept only; it may (and in practice often will) coincide with a physical host or network domain, but it is not required to do so. Several domains may co-exist on a single host, or a domain may be spread across multiple hosts. See also ‘domain process group’ below. Domains are independent of each other, but may exchange configuration data. Domains may be entirely logical divisions, or they may be physical, e.g. by host or network address. A combination is quite in order.
Component	A component is a single process, multi-threaded, one of several making up the system. It co-operates with other components by exchanging messages with them. A component has a request message queue that may be accessed by other processes. A component provides services, which are invoked by placing messages (requests) on its inbound message queue. It may therefore be thought of as a server. Components can be distributed across the infrastructure. A component instance (i.e. a running component process) resides on a particular host. In most cases, a component instance is attached to a domain; however, in the case of the Deployment Manager component, all domains communicate with it. A component instance can handle a single domain, or a fraction of a domain in the case of a domain being spread over several hosts. Components can be distributed across the infrastructure, and relocated in real time. A component may be a mirrored / remote component. Requests are delegated to the remote component.
Domain Process Group	A domain process group is an instance of the overall system, i.e. the framework plus the payload components. It is the set of all the running component processes that comprise a single instance. The processes may be running on one or more hosts. They are launched and monitored by one or more of their number (the host manager and the domain manager) and may be stopped and restarted by them after a failure.
Deployment	Deployment is the propagation of system configuration information between domains. This configuration information objects are of two kinds: Set-up parameters for the framework itself or for any of the application components Object Definitions used by the application components. The objects to be deployed are allocated version numbers by the framework; these are used by the system operator to designate objects to be deployed Deployment is under the control of a system operator, but is automatic once the data to be deployed and the destination domain have been specified. With the aid of the Web server plug-in component, data may be deployed across an enterprise intranet or the Internet

Framework Architecture

These are the component processes that make up the J-FORCE® framework architecture. Each is more or less independent of the others, although in some cases the amount of useful work that can be done in their absence is very restricted.

The J-FORCE® components are related below :

Host Manager	The host manager monitors and controls the life cycle of: Domain managers or other components on a given host Host managers on other hosts There is exactly one host manager on any host where any components reside Host managers communicate with their peers on other hosts, for propagation of requests involving the starting of processes on remote hosts. The set of host managers in an instance of the framework are all known to each other; they sort themselves into a ring structure for the purpose of monitoring and start-up. Thus each host manager monitors at most one other host manager. Host managers are monitored by other host managers if there are more than one; they are also monitored by the domain managers on their local host, and may be restarted by them in the event of failure. The host manager is the initial connection point for external clients. From it the clients obtain a list of references to available domain managers; after connecting to a domain manager clients can then obtain references to other components, with whom they may then communicate directly.
Domain Manager	The domain manager is the ‘master’ process of a domain process group from the viewpoint of process monitoring and automatic recovery. The main purpose of the domain manager is to manage the life cycle of all the underlying components in its domain, as well as its local host manager. It monitors all of its subordinate components via the ‘heartbeat’ mechanism. Another job of the domain manager is to supply references to components in the domain, for use by other components or external clients. The domain manager restarts components directly if they reside on the same host as the domain manager, or by means of a request to the host manager if the component resides on another host The domain manager propagates significant requests and events to other domain managers, via the deployment manager (another component, discussed below). One domain manager is configured to manage the life cycle of the deployment manager. The deployment manager is a component that facilitates transfers of version information between domains. As such it is the logical parent of the domain managers, but from the viewpoint of process monitoring it is subordinate to one of them From the viewpoint of process management, the domain manager may spawn the host manager and deployment manager processes. However, in the hierarchy of domain requests, the domain manager is the logical child of the deployment manager. From the viewpoint of process management, the domain manager and the local host manager may spawn each other. In the hierarchy of domain requests, the domain manager is the logical parent of the host manager. The domain manager knows all the components necessary to construct a complete domain. This configuration information comes from the configuration manager (see below). The domain manager knows all the components necessary to construct a complete domain. This configuration information come either from the configuration manager or as start-up command parameters for ‘bootstrapping’, when no configuration manager is available, or when a corrupt configuration database is to be overridden.
Configuration Manager	The configuration manager is a ‘slave’ component process of a domain process group. The purpose of the configuration manager is to maintain the configuration registry database and serve up data values when requested. The configuration manager keeps release-dependent data up to date, by making requests of the deployment manager .
Deployment Manager	The deployment manager is a ‘slave’ component process of a domain process group, but only from the viewpoint of process management. In the hierarchical structure of components, the deployment manager is at the top of the tree, as it the only means by which systems of one domain may communicate with those of another. The deployment manager is a component that is started up by the domain managers for the particular host on which the deployment manager happens to reside The purpose of the deployment manager is to handle inter-domain data transfers and queries on behalf of the domains under it. It manages the release definition database The deployment manager is an optional component in the sense that the other components of a domain can continue to run without it; however, it is essential for migrations. The deployment manager’s configuration information includes a list of the domains that it is to manage and the whereabouts of their domain managers. All domain managers know where to find the deployment manager as this information is part of their essential configuration data.
Authenticator	The authenticator is a ‘slave’ component process of a domain process group. It maintains the permissions database, which contains records of users, user groups, privileges, and assignments of privileges to user groups. The authenticator determines what actions a given user has permission to perform. It supplies session ‘tickets’ that must be transmitted along with most server requests The authenticator, like the deployment manager, is an optional component in the sense that the other components of a domain can continue to run without it; however, not much useful work can be done in its absence.
Application	An Application is one or more ‘slave’ component processes of a domain process group. They communicate with the framework processes, and with each other, by means of the same message-passing mechanism that is used by the rest of the framework The framework has no knowledge of what the application processes do, or of the meaning of application-dependent data that the framework may transmit on the payload’s behalf. The framework’s job is to launch the application component processes at start-up, restart them on failure detection, supply the message-passing mechanism, and automatically make up-to-date configuration data available. Each of those servers may themselves be multiple-component; this aspect is handled automatically by the framework

Communication

The framework components communicate only by means of messages, which they are able to place on to the inbound message queue of the target component. A message may be a request made by the originator component for the target component (server) to perform, or it may be considered a result from a request, or an asynchronous event (e.g. alarm).

The format and meaning of a message is determined by the target process. A message will generally contain some means of identifying the originator, in case the action of fulfilling the request in the message generates further messages to be sent back to the originator.

The mechanism for sending a message is a CORBA interface. Each component will publish an interface that will include at least one ‘enqueue message’ remote function.

(There may be other CORBA remote functions too, supporting process monitoring and control; non-framework components must also supply these functions. These are discussed elsewhere, under the Process Monitoring Sub-system)

The message sending is unidirectional, with delivery on a best-effort basis, not guaranteed. This is supported by the CORBA ‘one-way’ request protocol, and allows for more efficient transmission since there is no reply to wait for.

This architecture simplifies process design in one respect: there is no need to worry about whether your request succeeded or not, or even whether it was received. Results received are handled by a different thread, and can be matched up to originating requests by request id. However, someone somewhere, perhaps further up the chain, is going to want to know when requests ‘disappear into a black hole’, i.e. no result received within a reasonable time. An example of this situation is when a service definition is composed of several parts that must be executed in order; the process executing them must wait for an indication of the success or failure of one part before it sends the requests to execute the next part.

A method of handling this problem is as follows: requests can be optionally tagged ‘notification required’; for these requests, a timeout item is placed by the request originator on an internal list. The thread processing the incoming results matches them up with timeouts and removes any matching timeouts. Another thread wakes periodically and checks for timeouts that have expired; it remove these from the list and posts a ‘failure’ message to the results queue, where it is handled by the results processor thread. Thus all requests eventually get an answer of some sort

The ‘notification required’ attribute and the mechanism to handle it will be supplied by the framework.

Timing out may be caused by failure of some component down the line, or by temporary system overload. The process monitoring system (q.v.) must carefully distinguish between the two cases; restarting components that are merely busy will compound the problem.

Queuing

The framework components each manage one or more message queues. Each component will have at least one inbound request message queue, and optionally one or more outbound message queues.

Messages are placed on a component’s inbound queue by external processes. No need can be seen at present for more than one inbound message queue.The inbound queue is processed by a number of consumer threads; these take the next available message from the queue and perform the required actions to satisfy the request. Outbound queues are for the use of the component when sending messages to other processes that may be temporarily unavailable. Messages may be grouped by priority or other criteria and thereby processed in a different order to that of their enqueuing.

Queues may be transient or persistent. Transient queues are memory resident. Persistent queues are database resident and processed directly via the DBMS.

J-Tools
J-Tools® is a general purpose suite of C++ Class libraries. The Class library encapsulates : strings, link lists, portable streams, queues, events, threads, Name Value Pairs (NVP) manipulation. J-Tools® is required for development of J-FORCE® applications, but may be used independently in other applications.

Application Components

Main entities that comprise an J-FORCE® application component are related below :

Queue	A queue is a means of storing data objects in some kind of list structure. Queues are normally first-in first-out (FIFO) but it is also possible to sort the queued items or to retrieve them in a non-FIFO order based on some attribute, e.g. priority. As well as the interface message queues describe previously, there may be internal queues for other purposes A message queue is a specialised type of queue, for queuing and processing messages. It is used to effect communication between two groups of processes or threads: the supplier, which places messages on the queue, and the consumer, which removes messages from the queue and performs some action based on the content of the message. A message queue is processed by a number of consumer threads; these take the next available message from the queue and perform the required actions to satisfy the request. Between processing activity the threads sleep on a condition variable and are woken by action of the supplier inserting a new message in the queue. Queues may be persistent; this is taken to mean that the queue lives in a (DBMS) database permanently, rather than being archived whole. This kind of queue may be shared among components if the need arises.
Sub-System	A sub-system is an internal part of a component that performs a well-defined function. A sub-system is encapsulated, will typically involve a class or set of classes, and will reside in a shared library. In general a different base class will be needed for each sub-system; there is no functionality that is common between sub-systems. The J-FORCE® sub-systems on offer are : Resource Pool Queue Cache Process Monitoring Logging.
Abstraction Layer	An abstraction layer is a kind of sub-system. Its purpose is to isolate or encapsulate some aspect of functionality that varies between platforms or between third party software suppliers. The abstraction layer hides the details of the differences and presents a common API to the component code. Abstraction layers come in several flavours, corresponding to the different platforms or external systems to which the component is interfaced. In any given system build, a set of single specialisations from a set of abstraction layer base classes will be used. Implementation is totally dependent on the different platform or external system which is being abstracted. The design of the base classes have nothing in common with bases classes for other abstraction layers. A different base class will be needed for each abstraction layer; there is no functionality that is common between abstraction layers. The J-FORCE® abstraction layers on offer are: ORB DBMS
Thread Policies	As a general principle, a pool of threads will be created for each main task. This saves the overhead of spawning many new threads, and also keeps the number of threads within reasonable and known bounds. The thread pools will be managed by the resource pool sub-system (q.v.) A dedicated tuning thread will gather thread statistics regularly and use them to evaluate overall thread pool performance. Threads will be spawned or killed as performance thresholds are passed. Threads may be scheduled to run in two ways: Within the scope of a single process As if a system process If all threads are set up for contention within the scope of a single process, they will be run only when all other threads within that process are blocked. This will still make the process as a whole more productive, in that threads blocking on a call involving system resources can yield to other threads. When threads are set up for contention within the scope the system, they compete with other system processes on an equal footing. This means that the process can gain a larger share of the entire system resources, and individual threads can be run on different CPUs. There are several possible ways of controlling the sequencing of thread execution: First in, first out. Round robin Timesharing Priority Threads may be allowed to proceed to completion, or until blocked, or may be pre-empted by a higher-priority thread.
Resource Management	Resource management is a code design philosophy covering memory allocation. Its intent is to prevent ‘leakage’ of resources such as memory, and also to avoid errors such as double deletion and de-referencing of invalid pointers. It is said to be entirely successful in the former objective when used consistently, and to go a long way towards avoiding the other error conditions mentioned. Resource leakage and invalid pointer errors are serious coding errors which will eventually cause a system to crash. They are particularly difficult and therefore expensive to debug, and in this case an ounce of prevention is indeed worth a pound of cure. The objectives are achieved by the use of ‘strong pointers’, which are objects containing pointers to allocated data; they also embody the concept of ownership, which can be transferred when copying the pointer. The allocated object is guaranteed to be destroyed when the owning object goes out of scope. For the method to work, the use of the ‘new’ operator and the ‘malloc’ function are prohibited except in constructors. Data objects which allocate resources (most in practice) must always be handled using the ‘strong pointer’ objects. Strict observance of these rules will guarantee that all allocated heap memory is eventually released, which should go a long way towards the elimination of memory leaks and associated problems. Any exceptions to the usage rules must be rigorously justified and thoroughly documented.
Component Class	The component class provides the core functionality for all component processes in the J-FORCE® framework. The basic function of the component is to manage a set of queues, each of which has a pool of threads to process queued messages. Any component will have one or more message queues; the number of queues and types of message handled will be determined in the derived classes. The purpose of the component class is to encapsulate the above functionality, plus the subsidiary function of communications from the master to the slave process for the purpose of monitoring; this functionality is provided by the process monitoring sub-system. To allow the use of this class by both the domain manager and the subordinate components, the process monitoring functionality will be only for the process that is monitored, not the one doing the monitoring. Monitoring components will implement that functionality separately.
CORBA/IDL	The function of the IDL definitions is to specify the CORBA interfaces between J-FORCE® component processes The IDL definitions are compiled to produce two sets of C++ source files, one for the server and one for the client. With the help of the ORB abstraction layer, the classes and functions that the compiler outputs can be used to communicate with remote processes. The interfaces defined are: Send a message to a component’s queue. Heartbeat request

High Availability

J-FORCE® supports High Availability (H/A) in it's framework design and operation. In addition J-FORCE® compliments Hardware vendor H/A solutions.

Traditionally, H/A has best been described as "the ability to fail over one node running a suite of applications to another in the advent of a hard physical error". The downside with this is that the time taken to "bring up" the application has always been too long much less the nuance of "fail back" when and if required.

Efforts have been made to reduce the "outage time" and continuation of service by hardware disk mirroring and network clustering. In addition, the concept of replication has been introduced to increase the "Up Time" and just render service availability to simple re-connectivity.

The cost of implementing H/A is immense. The business benefit sometimes can be overindulged and outweighed by the actual cost of implementation and on-going operational costs.

By default, J-FORCE® has built in H/A concepts that can either dovetail into a distributed, hardware specific solution or be used in conjunction with H/A software products and their solutions inclusive of the designate hardware configuration. Such H/A software products are MC Service-Guard™ from Hewlett Packard and Sun Cluster™ from Sun Microsystems. J-FORCE® has been certified against both of these products.

Operating Systems / Development Kits


J-FORCE® has been designed and built for UNIX™ and Microsoft Windows. Current implementations of UNIX include SUN Solaris, HP/UX and Linux. Windows includes NT, 95, 98 and 2000. The CORBA, ORB dependencies for J-FORCE® are subject to the same deployment. J-FORCE® has a extra dependency of the J-Tools® product from JEDIS Limited. J-Tools® has been ported to the afore mentioned operating system platforms. For building J-FORCE® applications an C++ compiler must be procured to the target operating system. In addition an CORBA, ORB product must be procured. J-FORCE® bundles OmniORB™ from ATT Research as it's default ORB. However GNU licensing must be strictly observed if deploying a commercial J-FORCE® based product using the OmniORB™ product; refer ATT Research at http://www.uk.research.att.com/omniORB/omniORB.html for more details.

JEDIS Limited
PO Box 7005
Wellesley Street
Auckland
New Zealand

www.jedis.co.nz

Email: angela@jedis.co.nz

Ph: + 64 21 330 660
Fax: + 64 21 218 0660