Home
Nowadays batch applications are still a key piece of software in most organizations because a widely amount of business processes are amenable to batch processing. While online systems can also function when manual intervention is not desired, they are not typically optimized to perform high-volume, repetitive tasks. Therefore, even new systems usually contain one or more batch applications for updating information at the end of the day, generate reports, convert file formats, general asynchronous processing, and other non-interactive tasks that must complete reliably within certain business deadlines.
Focusing on this concept, we designed and developed an application that can cover the most common needs in an easy way. Simple Batch Manager (SBM) is a simple yet powerful batch management solution for enterprises environments of any scale, from small organizations to mid-size or even large. Their mission is to be the backbone of all your batch process needs.
The solution aims to manage the execution of processes that do not require interaction with the user. To do this, a Windows service was created with the ability to act as a launch pad. The service contains a repository of special components that run on a SandBox in a timely manner. These special components can be developed to cover one or more specific functionalities, or you can also reuse the already included generic ones.
Generally speaking, the product comes ready as an out of the box solution, which means that after a successful install, some generic functionality is already included and ready to use with minimal configuration. Of course, it depends on the complexity of your tasks, but the generic components can be configured quickly, providing some common services such as a bulk mail service (send / receive), a basic ETL (extract, transform and load), a DB command executor, and finally an OS command executor.
In short, just using the components included, you can for example: read a data source like an SQL DB and then write an Excel XML report file with the basic ETL component, create a new folder in a specific place and copy such Excel file on it from the current location with the OS command executor, then send it via e-mail with the mail service component, etc.
SBM is not a framework or API of any kind. Is not a language extension or a new development paradigm.
SBM is an application (Windows service), it is a complete solution to manage, execute, control, log and catalog your batch service needs. Is an integration software and a component oriented Middleware. You can develop any batch program for SBM using standard tools such as the C# or VB language (like others) and the .Net Framework, following only some simple rules. You can also use the generic "ready to run" services already included. You should install the application with the installer taking into account all the prerequisites, and then you can configure it with your preferences. And that's it, you're ready to use SBM.
As a definition and extending the bath program concept, each component that can be run under SBM can be seeing as a service. This kind of service should be specific, with a clear purpose, with low or limited complexity, always keeping the resources consumption as low as possible, and must have a maximum time interval for its execution. If your services are designed with this conception in mind, you can build a wide range of common service blocks that can collaborate with each other or serve many diverse application or systems.
After successfully implements some services, you probably realize that a few could be widely used, transforming it into a common or generic service. A clear example of these generic components are the generic services already included with the solution. Thus, you soon have a collection of generic services and specialized services also. These last one are services which implements certain tasks or operations that are specific or unique in your organization, and in many ways are related to a particular resources and environments.
As more and more services are incorporated, the ecosystem of services becomes more complex. Because of this, you must prioritize the development of services that can be shared with many applications or systems, making it reusable. The more reusable your service is, the more applications or systems could take advantage of that service. Since this type of service has some specific behavior and / or uses specific resources and / or follows certain common business rules, it must be considered different from a generic service. From the organizational point of view, we call this type of service a "standard service". In short:
- Generic Service: general tasks, highly decoupled from resources / environment
- Standard Service: common business rules, low coupled from resources / environment
- Specialized services: specific business rules, highly coupled from resources / environment
This set of generic, standard and specialized services makes up the so-called Service Federation Concept. These Federation of Services can be consumed on demand within the organization, or scheduled to run at specific dates.
SBM was designed to extract the maximum potential of this Services Federation concept, keeping a complete catalog of services and meeting all requirements of on demand or deferred execution. Having at the same time, the control and tracing mechanisms to follow every event and execution result of each dispatched service.
The solution includes the implementation of a specialized Windows service to manage the execution of assemblies or specific components (services), two Web Services that will allow interaction with the solution, and a dedicated database. The two web services will establish the ways to invoke, from different applications or clients, the offered services and later obtain the corresponding result notification or status information. These applications must be previously registered with the solution to be able to operate, in order to instantiate these services, they must provide an internal code that identifies it, in addition to a private key or token.
The first Web Service named SBM_QUEUE (SBM_Q) will be used to manage service requests. Each application requesting a particular service will invoke it by specifying the required service, a private identifier of the requesting application or owner, and finally the corresponding parameters. The WS SBM_Q will validate the request and, if successful, enqueue the request by returning the process ID assigned to it.
The second Web Service named SBM_QUERY (SBM_QRY) will be in charge of supplying information of both catalog (the services available for a particular application) and the state of the processes ordered and queued, for example, for a particular process identifier, or for a range of dates.
These two Web Services make up the SBM APIs, and are the only mechanism available to interact with the solution from other applications.
The database will be in charge of persisting all incoming and outgoing information, in addition to the one generated by the SBM itself.
Finally, the SBM executes the services that are required by the allocation of the necessary resources in a specific execution space, taking into account its architecture (32 or 64 bits) also. Once the service is started, it is monitored to know its status, in addition to watching its execution time, if so established. Additional to this, each service must have the necessary resources to carry out the task or tasks that are its own. For example access to a mail server, a specific network address, database, etc, that is, any resource that the service may require within the environment or domain in which it is running.
Due to isolation, security and scalability issues, it is advisable to have two separate deployment environments. In the first one that we can identify as the application server, it can contains the web server that supports the Web Services and the Core of the solution as a Windows service. In the second, a Data Server with the solution database.
Everything that can be executed in the SBM is a service, and that is the unique unit of execution. Each service can be as simple or complex as needed, although as defined, it must be specific and preferably of low complexity. A service thus established is packed in a component that will run under the architecture of the SBM. In short, a service should have the following characteristics:
- Specific: does a discrete task or set of simple tasks
- Clear purpose: solves a specific problem or need
- Limited complexity: it must be simple, if it is too complex it may have to be divided
- Low resources: you must use as few resources as possible (RAM, CPU, etc.)
- Limited execution time: must have a maximum execution time
Each service is executed as a single, autonomous process, nevertheless it is possible to create dependencies with other services. The concept of dependency establishes a parent - child relationship, whose sole objective is to define precedence and thus avoid the execution of a dependent service, if the parent did not execute successfully. In this way, a simple conditional execution relationship between services can be established.
All the services invoked can be executed at the same time on demand, even the same service, establishing an service execution pool that can grow as much as necessary, or resources are available in the SBM execution environment. However, a particular service may be classified as single-execution, preventing more than one instance of that same service from being run at the same time.
Each service then exposes a set of properties that determines its behavior and the handling that the SBM will give when it is instantiated. These properties are:
- Execution model: If it supports simultaneous instances (parallel execution) or not
- Platform: Indicates whether it is 32 or 64 bits
- Relationship: If it have dependency on another service
- Time limit: maximum execution time of the service
To run a service, SBM must receive a specific request that can be produced from either the web service, or from the internal system scheduler. While clients makes requests and queries directly on the web services arranged for this purpose, the scheduler works enqueueing directly the services that must be executed, according to their planning.
To configure the SBM operation, the launch module is responsible for reading the parameterization of the system and set it verifying that it complies with ranges and preset values.
The core of the system consists of three main modules, the service pool manager being executed, the scheduler and the service provider or dispatcher.
The Service Pool Manager has the purpose of maintaining and controlling the set of services that are running. This set of services represents records that the system will keep updated. Monitoring the state of the service pool allows the control of resources in use (number of running services, processing time and memory) as well as detect abnormal termination events during the execution of each one, updating also the system log.
Part of the SBM's responsibility will be to run scheduled services. The Scheduler module has this purpose, which has a simple time-controlled event scheme, to determine whether any service should be executed. When any execution condition is met, the module made the execution request automatically and directly.
Finally, the Service Dispatcher runs the requested services, consider the resources availability and recording their evolution. It's responsibility is to execute the requested services creating the appropriate environment, considering also the own service restrictions and system resources. It's composed of a queue of pending processes and a result queue where the product of the executed service is saved.
A typical batch program usually reads a large number of records from a database, a file or a queue, then processes the data in some way and finally writes that modified data. SBM automates this basic iteration, providing the ability to process similar transactions as a set, usually in an offline environment, without any user interaction, and asynchronously. Batch and asynchronous processes are part of most IT projects and SBM provides a complete and robust control and running environment.
Business Objectives
- Running of periodic and on-demand services
- Running of concurrent or parallel processing services.
- Manually or on demand cancel and restart services
- Sequential execution of dependent processes
- Services monitoring and traceability
- Structuring related services in composed units called Jobs
Technical Objectives
- Developers use the SBM model: they focus on the logic of the process and what needs to be resolved, using their usual development tools.
- There is a clear separation between infrastructure, execution environment and the batch process or service.
- The services presents an autonomous implementation model from SBM
- The model provides a common set of interfaces and methods to request service executing and information from clients
- The solution provides a set of basic generic services ready for use.
- Easy to configure, customize and start up.
The following are a number of key principles, guidelines and general considerations to observe when building a service.
-
A batch architecture typically affects online architecture and vice versa. It is necessary to design with architectures and environments in mind, using common building blocks as often as possible.
-
Simplify as much as possible and avoid the construction of complex logical structures applied to the same service. Reuse before redoing.
-
Process the data as close as possible to the physical location where they reside.
-
Minimize the use of system resources, especially I / Os. Perform as many operations as possible on the internal memory.
-
Review the application's I / O (analyze SQL statements) to ensure that unnecessary physical I / O is avoided. In particular, we must look for the following four common defects:
-
Reading data for each transaction when data can be read once and cached or stored in a workspace.
-
Reread data for a transaction where the data was previously read in the same transaction.
-
Causing unnecessary table or index scans.
-
Do not specify key values in the WHERE clause of an SQL statement.
-
-
Optimize processes. For example, if you need to show the result of a calculation or summarize some data, do it if possible when the data is processed initially, avoiding to go through that data again at the end.
-
Program services observing good security and development practices. The system is as weak as the weakest of the implemented services.
-
Always assume the worst with respect to data integrity. Insert proper checks and validate records to maintain data integrity.
-
Implement checksums for internal validation where possible. For example, flat files must have a trailer record that indicates the total records in the file and and appropiate key fields.
-
Plan and run stress tests as early as possible in an appropriate environment with realistic data volumes.
-
Plan a comprehensive and consistent backup policy.
To help design and implement batch systems or services, designers and programmers need to identify basic blocks and patterns to implement. Business logic must be broken down into a series of steps that can be implemented using the following standard building blocks:
-
Conversion Applications: For a given file, perform the required conversion to another format for processing.
-
Validation applications: Validations ensure that all input and output registers are correct and consistent. Typically, validation is based on file headers and footers, checksums and validation algorithms, as well as log level cross checks.
-
Extraction Applications: Reads a recordset from a database or input file, selects records based on predefined rules, and writes the records to an output file.
-
Extract / Update Applications: An application that reads records from a database or an input file, and makes changes to a database or an output file driven by the data found in each input record.
-
Processing and Updating Applications: An application that performs processing on input transactions from an extract or a validation application. The processing will usually involve reading a database to obtain data required for processing, potentially updating the database and creating records for output processing.
-
Output/Format Applications: Applications reading an input file, restructures data from this record according to a standard format, and produces an output file for printing or transmission to another program or system.
In addition to the main building blocks, each application may use one or more of common tasks steps, such as:
-
Sort - A Program that reads an input file and produces an output file where records have been re-sequenced according to a sort key field in the records. Sorts are usually performed by standard system utilities.
-
Split - A program that reads a single input file, and writes each record to one of several output files based on a field value. Splits can be tailored or performed by parameter-driven standard system utilities.
-
Merge - A program that reads records from multiple input files and produces one output file with combined data from the input files. Merges can be tailored or performed by parameter-driven standard system utilities.
The basis of any batch system is the processing strategy. Among the factors influencing the selection of this strategy are: estimated data volume, concurrency with online systems or with other batch systems, available time process windows, etc.
The services or batch processes must be incorporated into the SBM catalog so that they can be used. Once configured, they are ready for management within the system architecture.
The services listed and enabled in the catalog can be invoked from two different sources: from the scheduler and / or invoked from the web service.
Whatever the origin of the request is, if it is viable, SBM will proceed to queue the request by creating a record in the queue of pending requests. This new record have a unique ID that will act as a handler.
SBM reviews this queue of pending requests at regular intervals, taking each of them in strict order of arrival (FIFO: first in first out). When the application take one record, it is moved to the table of dispatched records, to begin with the procedures tending to instantiate the requested service and parameterize it as appropriate.
If all the prerequisites are met, then the service is instantiated in an exclusive execution thread, passing the parameters that could be associated with it. Otherwise, the SBM logs the detected problem in the same record of dispatched services table.
At that moment, the service start its execution being monitored by the SBM. This monitoring function is intended to ensure that it does not exceed the maximum execution time, or suffers a fatal error that produces an abrupt termination. In a case of an exceeded time limit, the system cancels the service by killing the process in the assigned thread, and then logs the newness in the table of dispatched services according to the handler of the service. A similar effect occurs if a client request to cancel a specific running service. In case of an unexpected termination, the system also records the anomalous event with the information that it can collect in the same table, according to the handler that corresponds to it.
But if, on the other hand, the service ends normally and within the expected time, it must report to SBM the final status and result, persisting those data in the table of dispatched services also, in its own record.
Is also possible check the status of any service invoked from any client application, as well as its result with the methods exposed by the corresponding web service. Either individually with a specific ID, or with other concrete selection patterns.