XAF allows creating Enterprise Systems quickly – this is the thing which is especially admired by developers. With all the wealth of "out of the box" functionality, questions of "three-tier" architecture in the current version of XAF have still been worked out poorly, although the possibility of its implementation is provided by the framework.
Below we’ll show how to implement one of these features: to increase the reliability and efficiency of the execution in the system of time-consuming operations with the possibility of scalable performance.
Statement of the Problem
Reliable execution of long-running operations is one of the most important tasks for the Enterprise Systems. This is especially true for the operations and reports with the formation time of more than twenty minutes. Not only that such operations are blocking the client system, locking users to the standstill. The reliability of this scenario is also questionable: the client computer may crash or lose network connection during the hours-long report generation and generated report will be lost, and the time wasted.
Such resource-intensive operations are usually performed on high-dedicated servers: RAM is often not enough to complete the generation of a report or perform the calculations.
Another approach is the delayed generation of resource-intensive reports on the schedule with the following sending of the results to users. For example, at night you want to run the report generation automatically so that the managers are able to analyze prepared reports in the morning.
How to solve this problem?
The idea of Xafari Server
During the creation of business systems of automation, we were constantly faced with the need for asynchronous execution of calculations and deferred reporting.
We needed to perform calculations and create XAF Report asynchronously and preferably on a dedicated server. It was decided to develop a full service for XAF applications capable of generating in this scenario.
Initially, we tried to implement these services using regular module «XAF Workflow» - but it was not very comfortable due to the instability of the WWF service and its administration.
Using Devexpress Report Server also proved to be impossible: it is a universal product which is aimed at the solving of common reporting tasks. It is not intended for the work in the XAF-application.
The first step that we have planned to implement is to create the report server for Xafari (XAF) applications.
It should be noted that within the Xafari framework there is a Xafari Report Module which significantly extends the functionality of the standard XAF Reports Module. Thus, within Xafari Reports the developer may determine:
- the parameters of any structure and complexity
- arbitrary data source
- algorithm for populating the Database (implementation of the algorithm can be substituted for by customizing the system at the customer)
- a variety of templates for one report (XtraReport, Excel, PivotGrid, Files, View, Spreadsheet, RichEdit, etc. - the list of templates can be modified and extended with the deployment of the system at the customer)
Therefore, we needed a report server just for our "Xafari Reports".
On the analysis and discussions to solve problems in an asynchronous/deferred reporting, we decided to implement a universal solution "Xafari Server".
Implementation
Thus, it was necessary to develop a Xafari Server - a separate service that handles the tasks in the message queue and stores the results in a XAF application database.
XAF has a carefully elaborated and flexible architecture and allows you to find unconventional solutions due to the presence of sufficient extension points. However, we had to spend a lot of time and effort to get the implementation of the operations on the Xafari Server from the user's account, as well as to provide a real multi-threaded server operation. The details have been given below.
The ease and convenience of using XAF is explained by a unified approach to the application development. We tried to keep the same principles while using the Apps Server. When writing the algorithm for calculating or report design an application developer should not greatly wonder where the code will run: either on the Apps Server or on the application on the workstation. If a customer has a Xafari Server, some operations can be transferred to it.
It was necessary to teach the client application to interact with the Xafari Server. To do this, we had to implement a message queue to which customers generate new tasks (messages) which are then processed on the server. Xafari Server is a separate service that handles the task in the message queue and stores the results in a XAF application database.
During the implementation of the project the following problems have been solved:
- Distributed message queue
Undoubtedly, MSMQ, RabbitMQ, and other common services could have been used. However, this would have considerably complicated the deployment and configuration. That’s why we have implemented a simple "built-in" version of MQ, and we also provided an API which makes it possible to connect and use the external service posts. - Processing of messages in the queue and saving the results in the database
We chose one of the simplest possible message flow, namely, through a common XAF application database. This greatly simplifies the overall architecture and makes it more transparent for XAF developers and administrators. - Full support of XAF and Xafari functionality
It was also absolutely essential for the Xafari Server to meet the important requirements: all the existing reports and business operations could be performed on the Xafari Server without any modifications or additional requirements for their coding. - Multithreading Xafari Server, pool of database connections
Great efforts had been taken to meet this requirement. In this respect, the platform XAF has quite a complex implementation - but the task itself was not a simple one. Many thanks to those who create and support sections of tickets and detailed documentation on the site Devexpress, which are really useful tips and advice. - Integration with the XAF security system and Xafari
Another rather difficult task within XAF application is to begin performing of the operation from the user, who had formed a task to the message queue. This is very important because the user who is running the report should get the data in the report in accordance with their permissions and privileges defined in the system security. - Interaction between application and message queue
We had to develop an API to work with the message queue in the client application to post messages to the queue and get results. In this case, we did our best to provide a possible substitution of the functional if necessary (for example – a switch to the use of RabbitMQ). - Processing messages on behalf of the user
The second part of the API for the work with the message queue, which is the part for reading the messages on the Xafari Server, allows processing a message on the Xafari Server on behalf of the user who initiated the task.
The ease and convenience of using Xafari Server is explained by the use of all the advantages of XAF, storing messages from the queue in the database, as well as storing the results of processing messages. This allows you to make the work of an application programmer absolutely "transparent".
On the whole, we have managed to fully achieve our objectives.
There had been doubts whether we would be able to provide an acceptable performance for its own implementation of the queue. The testing has shown that the performance of the production MQ services, of course, could not be reached. However, the obtained results fully satisfy the needs of current tasks. For example, on the average workstation, the performance message queue was more than 1000 messages a second, which is more than enough for our target scenarios. We also managed to solve the problems of locks set by numerous parallel handlers.
Message queue
We would like to highlight the main advantages of the message queue, here we are giving the link to the post by Roman Kononov "Message Queuing". This article describes the reasons for which the message queue is vital for any enterprise application architecture. Here is the abbreviated description:
- Low Coupling - the queues determine communication interfaces that enable applications to be independent of each other, only the message format is determined
- Redundancy – queues provide an opportunity to minimize the wasteful use of hardware resources (memory) for storing "extra" (unprocessed) data
- Scalability - queues allow to distribute information processing on different servers and dedicated services, thus making it easy to increase the capacity of the system by connecting additional nodes/message handlers
- Flexibility - message queues can act as a kind of buffer for storing data in the case of peak load, thus mitigating the burden on the data processing system and avoiding treatment failure
- Reliability - message queues allow you to isolate processes from each other; with the "fall" of the message queue handler, unprocessed messages can be processed later after the recovery of the system
- Guaranteed delivery - the use of a message queue ensures that the message is delivered and processed in any way (as long as there is at least one handler)
- Guaranteed delivery order – queues services, if necessary, help to ensure that the data will be processed according to the determined order
- Buffering – the process of adding to the queue takes place as fast as the message queue (and not the message handler) is able to perform this operation.
- Understanding of data stream - message queues allow identifying the most loaded channels and data streams in the system, which allows you to manage and optimize the load redistribution
- Asynchronous communication - message queues provide an opportunity to carry out data processing and calculations without blocking the system which sent a message to the queue, thus making the system more interactive
With the release of Xafari.MQ and Xafari Server all the developers, administrators and customers of the systems on the platform Xafari will be able to use these advantages of message queues.
The first release in Xafari x06
We are releasing Xafari Report Server in Xafari x06 version
The user creates a new report which is processed on the server and stored in the application database. The user can view the saved report at any time.
In this scenario, we had to further implement the ability to save the results of the reports in the database for long-term storage, as well as services that clean the storage.
Expansion plans and development
Next, we are planning to implement the following projects:
- Extended Xafari Server for asynchronous execution of business-operations
- Servers configurator to monitor their work
- Administrative module for the management of the queue of Xafari Server
- Message handling services on schedule
- API and services of notices and reminders
Follow our publications.
We will welcome feedback and we will be really happy to discuss various options for the use of the services, as well as development priorities.