 |
Qorus Integration Engine® Enterprise Edition 6.0.16_prod
|
|
Qorus Integration Engine® Enterprise Edition 6.0.16_prod
|
Back to the Developer's Guide Table of Contents
All Qorus integration objects are described by YAML metadata produced by our IDE, which is available as a free (as in free of charge and also open source) extension to Microsoft Visual Studio Code, a multi-platform editor from Microsoft. The Qorus extension is called Qorus Developer Tools and can be installed directly from Visual Studio Code.
User services are like named and versioned API sets that can be live-upgraded. User service methods are available to all workflows and jobs (and to other services) and can be automatically exported to other applications through lightweight web services (RPC protocols and REST) through the HTTP server (in fact, this is the default behavior, but can be inhibited by setting the internal flag on the service method).
Additionally, services can also have one or more background threads, and therefore do not have to wait for input data to perform some processing.
Services can also be called from the command-line using the qrest program as follows:
unixprompt% qrest put services/<name>/<method>/call args=...
Services are defined in YAML in files defining service metadata including an attribute giving the file name of the service class source that implements the method logic.
Services are loaded from the database into their own program objects and have their own internal API. All user-defined services must be of the type "user". System services are delivered with Qorus and should not be modified, as this is likely to affect system stability.
The following diagram illustrates the a subset of the attributes of a service.
Service Metadata
| Key | Mand.? | Description |
| Service Type | Y | The type of service: system or user |
| Service Name | Y | The name of the service; generally considered unique |
| Service Version | Y | The version of the service; only one version of a service can be loaded at a time |
| Service Patch | N | A string that describes the patchlevel of the service |
| Service Description | Y | The description of the service; supports markdown in the UI and the IDE |
| Service Author | N | The author of the service |
| Service Library | N | Library objects providing additional code to the service |
| Service Mappers | N | A list of data mappers that the service can use |
| Service Value Maps | N | A list of value maps that the service can use |
| Service Language | Y | The language of the service's source code |
| Service Source | Y | The source code to the service |
| Service Methods | N | The methods implemented by the service's source code and/or base classes |
| Service Configuration Items | N | Configuration for the service |
| Service File Resources | N | File resources for the service |
| Service API Manager Configuration | N | API manager configuration |
| Service Autostart Flag | N | The autostart flag that indicates if the service should be started automatically |
| Service Remote Flag | N | The remote flag that indicates if the service runs in its own process or not |
| Service Finite State Machines as Method Handlers | N | Finite state machines handling service method calls |
| Stateless Service Kubernetes Initial CPU Requirements | N | (Enterprise Edition only) The initial CPU allocation for stateless services running in Qorus in Kubernetes |
| Stateless Service Kubernetes Initial Memory Requirements | N | (Enterprise Edition only) The initial memory requirement for stateless services running in Qorus in Kubernetes |
| Stateless Service Kubernetes CPU Usage Limit | N | (Enterprise Edition only) The CPU usage limit for stateless services running in Qorus in Kubernetes |
| Stateless Service Kubernetes Memory Limit | N | (Enterprise Edition only) The hard memory limit for stateless services running in Qorus in Kubernetes |
| Stateless Service Kubernetes Autoscaling Min Replicas Value | N | (Enterprise Edition only) The minimum number of replicas that will be started when Qorus is running under Kubernetes for stateless services only |
| Stateless Service Kubernetes Autoscaling Max Replicas Value | N | (Enterprise Edition only) The maximum number of replicas that will be started when Qorus is running under Kubernetes for stateless services only |
| Stateless Service Kubernetes Autoscaling CPU Target Value | N | (Enterprise Edition only) The percentage CPU usage goal for Kubernetes for all pods for stateless services only |
| Stateless Service Kubernetes Autoscaling Memory Target Value | N | (Enterprise Edition only) The memory usage target for Kubernetes for stateless services only |
| Service Stateless Flag | N | (Enterprise Edition only) The stateless flag for the service; indicates if the service is started externally to Qorus can can run in multiple processes |
| Service Modules Parameter | N | A list of Qore-language modules to be loaded into the service's Program container |
There are two types of services:
system: for system services delivered with Qorususer: user services written / configured by Qorus developersThe name of the service; the name and version together are unique identifiers for the service and are used to derive the serviceid (the single unique identifier for the service; it is generated from a database sequence when the service is loaded into the system via oload).
There can only be one service of a given type and name loaded in the system at any time, so the tyoe abd name of a service together also make a unique compound identifier for the job, however since system services are stable, and user services should avoid using names used by system services, the name of a service is often used as a unique key as well.
In case a user service might have the same name as a system service, the user service would be inaccessible with APIs (such as the REST API) where the service name is used as a unique key, therefore it is advised to never name a user service with the same name as a sytem service.
Version string for the service; the version is informative; only the latest version of a service can be loaded in Qorus at any time, so services are referenced by name generally, and sometimes by type and name.
A string "patch" label which can be used to show that a service was updated while not affecting the serviceid.
patch value can be updated without affecting references to other objects; the unique ID for the object is not updated when the patch value is updated, however since service names are already treated as unique, the patch attribute is not as useful as in other objects but is still included in services for consistency's sake.Description of the service; accepts markdown for formatted output in the UI and IDE.
The "author" value indicates the author of the service and will be returned with the service metadata in the REST API and also is displayed in the system UI.
Services support library objects that provide additional code for the service.
An optional list of mappers that are used in the service.
An optional list of value maps that are used in the service.
The programming language used for the service implementation.
The implementation of the service must be made in the programming language given by Service Language; the main job class must inherit one of the following classes depending on the source language: The source for a service is a class inheriting one of the following classes depending on the source language:
Service classes can have a constructor and classes can have static initialization, but please note that if the constructor or static class initialization requires features that are only available at runtime in Qorus itself, the errors raised in oload will cause service class instantiation or static class instantiation to fail.
To work around this, put all initialization requiring runtime support in Qorus in the init method of the service. This method is only called when the service is initialized by Qorus at runtime.
The service constructor must take no arguments.
Static class variables are initialized when the class is loaded which is also performed by oload, therefore if any static class variables have initialization code that requires Qorus functionality, this can cause oload to fail, therefore it's recommended to put such static initialization in the class's constructor() method instead as in the preceding example.
Service methods define the user-visible interface of the service. The logic in a service program is defined by the method code and any library objects loaded into the service's program container.
Service methods are the user-visible entry-points to a service and as such contain the logic of the service.
The following diagram illustrates a subset of the attributes of a service method.
Service Method Metadata
| Key | Mand.? | Description |
| Service Method Name | N | The name of the method; also a public method with this name must be defined in the source as well |
| Service Method Description | Y | Description of the method object |
| Service Method Author | N | The optional author of the method, in case it differs from the service's author |
| Service Method Read Write Lock Setting | N | must be either none, read, or write (default none), to determine how the service's RWLock will be grabbed before the method is executed |
| Service Method Internal Flag | N | If this boolean flag is set to True, then the method will not be automatically exported through the any network interface |
| Service Method Write Flag | N | (Enterprise Edition Only) If this boolean flag is set to True, then the method will be marked as a write method, meaning that external callers will have to have the CALL-USER-SERVICES-RW role to call the method if RBAC security is enabled (Enterprise Edition only) |
The service method name must be unique (although service methods may be overloaded in Java and Qore) and is used when calling the method or accessing it using the REST API.
A string giving the name of the author or authors of the service method.
The description supports markdown in the UI and IDE.
Each service has a read-write lock object (Qore class RWLock) associated with it. Methods can be tagged to acquire the lock for reading, writing, or not acquire the lock at all (default behavior).
This attribute can be used to provide safe multi-threaded access to internal resources. For example, if the service provides methods to read data and a method to reload the data from an external resource (such as a database), then the read methods can be tagged to acquire the lock in read mode, and the method to reload the data can be tagged to acquire the lock in write mode. This way, the reload method will block until all reads have finished, and the reads will block if any reload is in progress.
The read-write lock will be released safely when execution returns to the Qorus system, even if an exception is thrown in the method.
If the "internal" flag is set to True on a method, then that method can only be called internally. Any direct call to this method from an external source (such as through lightweight web-service protocols exported through the HTTP server) will result in the method not being called and an exception returned to the called instead. This flag should be set on methods that return unserializable objects that cannot be sent through network interfaces.
If the "write" flag is set on a method, then users without the CALL-USER-SERVICES-RW (for user services) or the CALL-SYSTEM-SERVICES-RW (for system services) or another permission that contains these cannot call the service method from an external network interface.
copy() service method with a Qore class-based service, name the class method _copy() (with a leading underscore), and give the method name copy(). The translation to the internal name (to avoid a conflict with the special class method copy()) is done automatically by oload and by the Qorus runtime. See the following example for more information. constructor() destructor() methodGate() memberGate() memberNotification() Services can declare configuration items as metadata to allow for the behavior of the service to be modified by auhtorized users at runtime using the operational web UI or the REST API.
Service configuration items are:
Service configuration items are designed to allow users to affect the execution of a service so that changes can be made by authorized users in the UI without requiring a change to development.
If the strictly_local flag is True, then the service job configuration item is local and the value for the configuration item cannot be set on the global level.
If the strictly_local flag on a service configuration item is False (the default), then the service configuration item is not local and the value can also be set on the global level.
Services can react actively to configuration item changes by using the following APIs to register a callback that will be called when a config item value is updated:
Service file resources are resource files that are declared with the "resource" tag in a service definition file and loaded into the database with oload.
These file resources can then be served automatically if the service binds a OMQ::AbstractServiceHttpHandler object with ServiceApi::bindHttp() to serve HTTP requests.
Each resource name is its relative file name; for example if the following resource is declared in a service:
resource: html/*
And the file html/extension.html is loaded by oload as a service file resource, then "html/extension.html" is a valid resource name, and could be set as the default resource (the resource that's served when no other resource is matched to a request) with AbstractServiceHttpHandler::setDefaultResource().
For example, if the code in the following examples is used to create the service HTTP handler object and bind the handler:
Then a request like the following is received:
GET /my-service/html/extension.html HTTP/1.1
The cx.resource_path value in the call to AbstractServiceHttpHandler::handleRequestImpl() will be set to "html/extension.html", which in this example would be a valid service file resource, therefore if the AbstractServiceHttpHandler::handleRequestImpl() returns NOTHING, this resource would be automatically served to the requestor.
Mixed text/HTML/JavaScript and Qore code can be automatically rendered and served if the resource has the file extension: "qhtml", "qjs", or "qjson". In this case the final rendered text is rendered and served automatically according to the rules described in WebUtil::StaticTemplateManager::add().
Summary of unrendered text format:
{% qore statement %} {{ qore expression returning a string }} {% foreach hash<auto> $h in ($ctx.list) { %}
<tr>
<td class="name"><a href="?rpath={{ $ctx.parent_url ? $ctx.rpath : "" }}/{{ $h.name }}" {{ $h.type != "DIRECTORY" ? "class=\"file\"" : ""}}>{{ $h.name }}</a></td>
<td class="text-left" width="40">{{ $h.type != "DIRECTORY" ? "File" : "Directory" }}</td>
<td class="text-right">{{ $h.type != "DIRECTORY" ? $h.size : "--" }}</td>
</tr>
{% } %}const ws_server = '{{"ws" + ($ctx.ssl ? "s" : "") + "://" + $ctx.hdr.host + "/" + UserApi::getConfigItemValue("websockets-root-uri")}}';Qorus's service template manager sets up the embedded Program object for templates with additional code similar to a service Program.
The following are additions and enhancements available to template processing and rendering when used with Qorus services:
"Qorus" (common to workflows, services, and jobs)"QorusServer" (common to workflows, services, and jobs)"QorusHasUserConnections" (common to workflows, services, and jobs)"QorusHasAlerts" (common to workflows, services, and jobs)"QorusService" (service-specific)"QorusHasHttpUserIndex" (service-specific)parse-options attribute of the service; it's not possible to set them with parse directives inside the template, as injected code (which is injected as source) would be injected with the wrong style. This means that the standard Qore style in template programs is "old style"; i.e. variables require the dollar sign (ex: $ctx) by default; "new style" (without dollar signs) is only used if set in the service's metadata. As setting parse options is not currently supported with YAML-based service metadata, all service templates are parsed with old style for services with YAML-based metadata.Use the Api Manager configuration to configure a Qorus service as a protocol handler using a supported API schema (currently SOAP and Swagger).
To configure a service as an API manager, you need to specify the schema type (soap to provide a WSDL or swagger to provide a JSON or YAML schema) and then the schema file. The IDE will then upload the schema file to the server, which will reply with endpoint information, at which time the IDE will allow you to associate a service method or a finite state machine to each API endpoint to provide the implementation of each API endpoint specified by the schema.
The schema file itself will be saved against the service as a file resource.
In the Enterprise Edition of Qorus, the stateless flag can be combined with an API manager configuration to implement a scalable microservice for handling APIs under Kubernetes.
Services can also define handlers for DataProvider events to allow for no-code / low-code definitions of event handling solutions in Qorus.
To configure service event handlers, the user must choose a DataProvider that supports events. Then each event can be associated with either a service method or a finite state machine to specify the logic to be used when handling the event.
Note that the following automatic variables are created when handling an event:
event_id (string): the unique name or ID of the eventevent_provider (DataProvider): the object providing the event sourceTo access the value of these variables in service methods handling events, you can use the following code:
If the "autostart" flag is set to True, then the service will be started automatically whenever Qorus is started or whenever it and all interface groups it is a member of are enabled.
"autostart" value is considered to be managed by operations, which means that once a service has been loaded into Qorus, if its "autostart" value is updated with the API, then those API-driven changes are persistent and will not be overwritten by subsequent loads of the service by oload.The "remote" flag indicates if the service will run as an independent qsvc process communicating with other elements of Qorus Integration Engine with a distributed queue protocol rather than internally in the qorus-core process.
When services run in separate qsvc processes, it provides a higher level of stability and control to the integration platform as a whole, as a service with implementation problems cannot cause the integration platform to fail.
There is a performance cost to running in separate qsvc processes; service startup and shutdown is slightly slower, and communication with qorus-core also suffers a performance hit as all communication must be serialized and transmitted over the network.
Calls to service methods in remote processes may not leave thread resources active when returning from the method; doing so will cause an exception to be thrown and for the thread resource to be cleaned up automatically.
Additionally, service methods running in remote qsvc processes cannot return non-serializable data (non-serializable objects, references, callable data types), and also their methods cannot accept these values as arguments. Furthermore, all HTTP or other protocol support implemented in remote services will be subject to round-trip network serialization to and from qorus-core.
"remote" flag must be set to False.The default for this option depends on the client option qorus-client.remote (if this client option is not set, then the default value is True).
The remote value can be changed at runtime by using the following REST API: PUT /api/latest/services/{id_or_name}?action=setRemote
remote flag is considered to be managed by operations, which means that once an interface has been loaded into Qorus, if its remote flag is updated with the API, then those API-driven changes are persistent and will not be overwritten by subsequent loads of the interface by oload.Finite state machines can be associated with service methods as the trigger to enable service methods to be implemented with a low-code or no-code solution. When a finite state machine is associated with a service methods as its trigger, the finite state machine is executed whenever the method is called, and any output data from the finite state machine is used as the return value for the method.
The arguments to the service method are available as input data to the finite state machine.
Finite state machines can be associated with service methods in the IDE as in the following graphic:
"Stateless" services are services that scale horizontally across many processes and potentially many VM containers.
Statless services automatically support a "soft failover" for service calls and accesses to service resources; if a stateless service program instance dies in a call, then another client is chosen automatically and the request is sent to a running client without passing the error back to the caller.
Stateless services can serve HTTP, REST, FTP, and WebSocket requests; see the class documentation for the particular handler class for more information on how they behave with stateless services.
When Qorus is running under Kubernetes, stateful sets are created and destroyed automatically for stateless services, and additionally the associated pods are scaled to zero when stateless services (or an interface group that they are a member of) are disabled.
Qorus determines that it is running under Kubernetes if processes are started in independent mode and the KUBERNETES_SERVICE_HOST environment variable is set.
Stateless services are always started externally to Qorus (for example by Kubernetes or OpenShift); they are assumed to always be remote, and additionally the autostart flag is ignored.
Stateless services are stopped when they are unloaded or reset; in such a case the external software can decide to restart them; normally they will be restarted when an unload or reset action is initiated by Qorus.
The initial CPU requirement for stateless services when Qorus is running under Kubernetes expressed as a floating-point number (ex: 0.5 = half a CPU).
This number is used by the Kubernetes scheduler to determine if a service can be scheduled; if no node has enough free CPU resources, then the service will not be scheduled and cannot be started.
The initial memory requirement for stateless services when Qorus is running under Kubernetes expressed as a string (ex: 250Mi or 2Gi).
This number is used by the Kubernetes scheduler to determine if a service can be scheduled; if no node has enough free memory, then the service will not be scheduled and cannot be started.
The hard CPU usage limit for stateless services when Qorus is running under Kubernetes expressed as a floating-point number (ex: 4.5 = four and a half CPUs). If the service exceeds this value, it will be subjected to CPU usage throttling.
The hard memory limit for stateless services when Qorus is running under Kubernetes expressed as a string (ex: 250Mi or 2Gi).
If the service exceeds this value when running, it will be terminated.
The minimum number of replicas that will be started when Qorus is running under Kubernetes.
This option is valid for stateless services only and also only takes effect when Qorus is running under Kubernetes.
The maximum number of replicas that will be started when Qorus is running under Kubernetes.
This option is valid for stateless services only and also only takes effect when Qorus is running under Kubernetes.
The percentage CPU usage goal for Kubernetes for all pods for the service when Qorus is running under Kubernetes.
This option is valid for stateless services only and also only takes effect when Qorus is running under Kubernetes.
The memory usage target for pods for the stateless service when Qorus is running under Kubernetes.
This option is valid for stateless services only and also only takes effect when Qorus is running under Kubernetes.
The "service-modules" option lists modules providing base classes for Qore-language services classes.
Modules declared like this will be loaded into each service's Program container, and their classes can be used as base classes for service classes.
Otherwise, a service is loaded and initialized any time a method of that service is called and the service has not already been loaded. In this case all the service's methods are loaded and parsed into the same program object, along with any library function, classes, and constants associated with the service. Any parse exceptions during service loading will prohibit the service from being loaded.
If a service has an init() method, this method is called as soon as the service has been loaded and parsed, and before any call to a service method is made. For example, if a service with an init() method is loaded because a call to another method is made, the init() method will be called first, and then the called method will be called and the value returned to the caller.
An exception when running the init() method will cause the service to be unloaded, and the exception will be returned to the caller.
If the service has a start() method, then it must also have a stop() method. The start() method will be run in a separate thread and is expected to run until the stop() method is called. The stop() method should signal the routine running in the start() method to exit.
When the start() method returns, the service is automatically unloaded. To start threads in user code in a service, use the svc_start_thread() or svc_start_thread_args() functions (in which case, to use these functions, the service must also have a stop() method).
init() method is explicitly called, this call is ignored, in the sense that, if the service is already loaded, no action is taken, and if it is not, the service is loaded and initialized, so the init() method is called as per the rules above. Therefore to simple ensure that a service is loaded and initialized, it is safe to call the init() method of a service at any time. Also note that the init() method may be safely called even if no init() method is defined for the service.Qorus services can implement generic HTTP request handlers by subclassing OMQ::AbstractServiceHttpHandler and binding it either to the global Qorus listeners or service-specific listeners using:
Using this mechanism, Qorus services can implement custom HTTP-based protocols or use one of the existing heler classes to export RPC services, REST APIs, web sockets server event sources, or even generic HTML user interfaces and Qorus web UI extensions.
The web UI can be extended with Qorus services by registering the extension with the following function:
The only argument of this function must be a class descended from OMQ::QorusExtensionHandler as in the following examples:
At least one service file resource is required: the bootstrap HTML page; this resource is set by calling OMQ::QorusExtensionHandler::setDefaultResource() as in the above example.
Resources must be declared in the service definition file with a service file resource declaration.
The following is an example UI extension service with example resources.
<h1>Hello World!</h1>
resource: index.html
hello-world-v1.0.py)hello-world-v1.0.qsd)REST APIs can be developed with Qorus services by subclassing the OMQ::AbstractServiceRestHandler class and bound with:
Consider the following examples:
This is a simple service declaration that exports a (very simple) REST API at "/example" that returns the string "OK" to the following requests:
GET /example HTTP/1.1: handled by ExampleRestHandlerGET /example/subclass HTTP/1.1: handled by ExampleRestClassQorus services can provide websocket server services by subclassing OMQ::AbstractServiceWebSocketHandler and bound with:
Consider the following examples:
The above example service will listen for all system events with waitForEvents() (polling every second for a service stop event) and serialize any new system events with JSON using make_json() and send to all connected clients using WebSocketHandler::sendAll().
Qorus services can implement a custom FTP server for receiving data by subclassing OMQ::AbstractFtpHandler and calling the following API on the resulting object:
Consider the following example services:
This service will create an FTP server on port 18002 by default and store any files received in the "/tmp" directory by adding ".tmp" on the filename.
The AbstractFtpHandler::fileReceived() callback could then be modified to create a workflow order instance for the file's data if necessary (for example).