Archive for the ‘Lapis Server’ Category

In the latest round of development of the Lapis Server, we’ve added the following functionality:

Hot Folders

you can now add a watcher to monitor file system folders. When files are created, modified or deleted, workflows can be instantiated automatically.
Files that can trigger the workflow may need to match a regular expression. For example, when a new video file is created, a different workflow is instantiated than when a pdf document is modified.
The workflow can be instantiated with a given set of properties. An additional property “lapis.workflow.watch.file” is set to the file name which triggered the workflow instantitation.
Below is an example of the watch.xml configuration file:

The server property “lapis.workflow.watch.config” can be set to point to an xml document configuring the folders watch as well as the workflows to instantiate.
the root element is the “watch” element and it has one attribute, named “runAs”, which defines which user to impersonate when instantiating workflows from hot folders.
The watch element has any number of folder elements. Each folder element has two attributes, “path” which defines the path of the folder to watch and “recurse” which contains a boolean (true or false) indicating if sub-folders must also be watched, recursively.
The watch element also has any number of graph elements. The graph element has one attribute, “path” which defines the path of the workflow file to instantiate relative to the graph store (defined by the server property “lapis.workflow.graphs”).
the graph element has any number of event elements. the event element has one attribute, “kind”, which defines the type of events that may trigger the instantiation of the graph. The kind attribute accepts the following values: “ENTRY_CREATE”,”ENTRY_DELETE”, “ENTRY_MODIFY”, corresponding to file creation, deletion and modification respectively.
the graph element has any number of regex elements. The regex element has one attribute, “match”, which corresponds to a regular expression which is compared to the path of the file which is being modified, created or deleted. If both the event kind and the regular expression match, then the graph is instantiated.
the graph element has any number of parameter elements. the parameter element has two attributes, “id” which defines the id of the parameter and “value”, which defines the value of the parameter. The parameters are passed to the graph during instantiation and set as properties (duplicates are removed).
A specific property, “lapis.workflow.watch.file” is also added to the list of properties and is set to the path of the file which was created, modified or deleted.

Group tasks

Now that we’ve added the capability of users to belong to groups, the next logical step was to add a group task.
just like a node can e assigned to a user, the node can also be assigned to a group or a number of groups. A group task starts its life without ownership. A user must acquire the ownership of the group task before they can complete it. Only members of the groups mentioned can complete the token on a group node. Group tasks no yet assigned to a user require the ownership to be set to a member of the groups listed in its groups property before they can be completed.
The group node type requires the below node properties to be set:
groups: a comma-separated list of groups whose members can acquire the ownership of the task and complete it.

Graph execution and node token filters

Graph execution and node token filters can now be created to accept or reject graph executions or node tokens respectively. This is used to filter lists of graph executions, which we now use in the web application.
Graph executions can now be filtered by id, name, owner or description.
Node tokens can be filtered by id, name, owner, type (email, group, sub, etc…) and description, group and you can expand this list by creating your own filters. I am sure more will get created as time goes by.
Because of this the web application menus have been changed slightly to separate listing of graph executions and node tokens.
/wfe/graphexecutions list the graph executions unfiltered whilst /wfe/user/graphexecutions automatically starts with a “owner” filter set to the current user. The same logic is applied to /wfe/nodetokens and /wfe/user/nodetokens.
Below is a screenshot of the web application showing this:

Email tasks

email task now accepts users, groups and email address in to, cc and bcc fields.

  • Groups are expended into a list of users,
  • Users are expanded into a list of email addresses and
  • Email addresses are added to the corresponding fields

Command tasks

The command task now is a bit more stable and also accepts the working directory parameter. I have fixed an issue with the stdout and stderr output streams.

Chrome extension and Ubuntu application

I was playing a bit with this more than anything and thought i would be easier to start browsing if there was a chrome extension opening the web site for me, so I built one for my development environment and another one for my production environment (I have started using the engine for my own workflows now – I figured if I want a truly fit for purpose workflow engine, I may as well use it for myself)

To make starting the engine easier, I created an Ubuntu desktop application launcher which starts the engine and also starts Tomcat, where the web app resides.
Below is the screenshot of the launcher file in ~/.local/share/applications

What’s next?

As always, I have a backlog to choose what I build next. I’m not sure what that will be but among the list is LDAP authentication module, an EZPack installer and improving working with “attached” files

Advertisements

The project is installed in a folder of your choosing, referred to as lapis_home. It is common to place to select /opt/fukoka/lapis as the lapis_home folder.
Inside the installation folder, we see a list of sub-folders, each with a specific purpose.

  • bin:
    the bin folder contains the command line utilities that help users and administrators interact with the lapis server.
  • conf:
    the conf folder contains your configuration files, for example the graphs (workflows) are located in this folder
  • docs:
    user guide, developer guide and administrator guide reside in this folder.
  • etc:
    the etc folder contains the configuration files relevant to the server. This would include the server properties file, the users’ XML repository, the jaas configuration file, the jaas policy file, the license file and the license signature file.
  • javadoc:
    contains the java documentation
  • lib:
    this folder contains the java archives which the server is relying on to function properly. The ext sub-folder is the location where you place your custom extensions to the server and 3rd party libraries
  • logs:
    this folder contains the log files generated by the server.
  • run:
    this folder contains files required or created when running the server. For example, the graph execution files are stored in this folder.

This sprint is almost over and I have been quite the busy guy. Below are the (important) features that have been added since my last post:

  • mail node: send emails from workflows
  • command node: execute external commands from workflows, such as perl scripts
  • encrypt/decrypt SMTP paswords
  • simplified graph execution serialisation to XML
  • password changes by users

Mail Node

I have created a core email task which connects to an smtp server using credentials stored at the server level. The server can then provide a session easily without details appearing in the workflows themselves. The password is encrypted in the property file and decrypted whenever a session is required.
Below are the connection details for my SMTP account (the password property provided is an encryption of the password):

mail.smtp.host=smtp.gmail.com
mail.smtp.port=465
mail.smtp.starttls.enable=true
mail.smtp.ssl.enable=true
mail.smtp.auth.enable=true
mail.smtp.starttls.enable=true
mail.smtp.auth.username=lpicquet
mail.smtp.auth.password=DTeF3oscP65GRg8sL9KiIVb+KIgohbQ3t

The mail task uses XSL to transform the workflow’s XML representation as HTML email. In the next few days, I made the email task to be able to have also a text/plain part, be able to accept custom content via a custom property, reference a file list of files to attach to the email.

The mail node accepts the following parameters:

  • onSuccess: name of the arc to follow if it all goes well
  • onFailure: name of the arc to follow in case of error. Can be the same arc as the sucess arc.
  • from: email address to send the email from
  • to: email address to send the email to
  • subject: the subject of the email
  • stylesheet.html: the URL of the stylesheet to use to create the HTML alternative content of the email
  • stylesheet.text: the URL of the stylesheet to use to create the text alternative content of the email
  • xmlContent: the text of an XML document to parse. If not present, the mailer will use the graphExecution’s XML representation.
  • fileList: a path to a file containing a list of files to attach to the email. The content disposition is set to “attachment”
  • relatedFileList: a path to a file containing a list of files that can be referenced by the email. The content disposition is set to “inline”. The content id is set to a named GUID, created from the path of the image so that it can be made easier to reference from HTML emails. The cid will therefore never change for a given path.

Command node

I also added a command task, which let’s you execute system commands. With it, it is now possible to execute perl, python, LUA scripts, or any other executable you want. This is starting to make this workflow engine quite powerful.

The command node allows the workflow to execute commands that you normally would type at the prompt. This enables you top interact with the scripts or other commands that you may have already written, such as Perl scripts for example.

The command may necessitate the use of an interpreter first. For example, the perl program helloWorld.pl is not to be called directly. Instead, it must be passed as an argument to the perl interpreter: “/usr/bin/perl helloWorld.pl”.

The command node takes the following parameters:

  • onSuccess: This arc will be followed when the command completes with a return code of zero.
  • onFailure: this arc will be followed if the command cannot be instantiated or if the return code is non-zero.
  • command: the command to execute, with its arguments
  • stdout: the file to create or append to with the content of the standard output
  • stderr: the file to create or append to with the content of the error output
  • stdin: optional argument to use if the program takes its input from standard input

Below is an example of a hello world program:

<command name="perl" owner="laurent" start="true"> <!-- credit where it is due: This is simply the best hello world script of the world. http://www.perlmonks.org/?node_id=329174 -->
 <arc name="failed" to="end" /> <arc name="done" to="end" /> <parameter name="onSuccess" value="done" /> <parameter name="onFailure" value="failed" /> <parameter name="command" value="/usr/bin/perl -w"/> <parameter name="stdout" value="logs/perl.out" /> <parameter name="stderr" value="logs/perl.err" /> <parameter name="stdin"><![CDATA[package Earth;sub Greet{
                 %_=('Y','~');$_='$;=!(Middle
              Earth.age~~~<Eart~~~~~~~~~~~~~h
           .age)?!(defined$ti~~~~~~~~~~~mez~~~On
         e[2])?!(push@time~~~~~~~~~~~~~~~~Zone,loc ~altime())?rotation?~~~~~~~~~~~~~q~~?The Worl ~~d?:q:[\w]::q=[\~~~~~~~~~~~~~~~~~d~a-f]=:q?..~~ ~~~?:q:.:;"42b3d3~~~~~~~~~~~~~~~~~~~~~728656c6c6f6 ~~~~~0277f627c64672~~~~~~~~~~~~~~~~~~~~~b3072796e647 ~~~~~~~42b3b3rg7d"=Ym~~~~~~~~~~~~~~~~~~~\$;~~*\;p~~~~u ~~~~~~~~~sh@_,$&;bless~~~~~~~~~~~~~~~~~~~~~~~~~$c~~~~~~~ ~~~~~~~~~o~ntine~~~~~nt~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~s=\~~~~~~~$~~~~~~~~~~~~~~~~~~~~~~~pangaea~~~~ ~~~~~~~~~~~~~~~;{l~~~~~~~~~~~~~~~~~~~~~~~~~~~~ocal@_;local$; ~~~~~~~~~~~~~~~~~="o~~~~~~~~~~~~~~~~~~~~~~~~~cean";$^A=(defi ~~~~~~~~~~~~~~~~~~~n~~~~~~~~~~~~~~~~~~~~~~~~~ed$continents)? ~~~~~~~~~~~~~~~~~~~(vec(~~~~~~~~~~~~~~~~~~~~~~$;, YYsplit(\' ~~~~~~~~~~~~~~~~~\',${\$;}~~~~~~~~~~~~~~~~~~~~~~)%3,YYsplit( ~~~~~~~~~~~~~~~~q??,$;)**2-~~~~~~~~~~~~~~~~~~~~~~(($;=Ytr/oa ~~~~~~~~~~~~~~~~eiu//)**2))=~~~~~~~~~~~~~~~~~~~~~~=28160)?q: ~~~~~~~~~~~~~~~~~.::q?!?:\'?~~~~~~~~~~~~~~~~~~~~~~\';}$^A=Ys ~~~~~~~~~~~~~~~~:\Q.\E:pack(~~~~~~~~~~~~~~~~~~~~~~\'h*\',j ~~~~~~~~~~~~~~~~~oin(q(),~~~~~~~~~~~~~~~~~~~~~~~grep{$_= ~~~~~~~~~~~~~~~~~~Ym,$,,}~~~~~~~~~~~~~~~~~~~~~~~split(" ~~~~~~~~~~~~~~~~~",@_~~~~~~~~~~~~~~~~~~~~~~~~~~[0])) ~~~~~~~~~~~~~~~~):e~~~~~~~~~~~~~~~~~~~~~~~~~~~gexe ~~~~~~~~~~~~~~~;$d~~~~~~~~~~~~~~~~~~~~~~~~~~~="s ~~~~~~~~~~~~~~ort~~~~~~~~~~~~~~~~~~~~~~~~~~<= ~~~~~~~~~~~~>,~~~~~~~~~~~~~~~~~~~~~~~~~~YY ~~~~~~~~~~~@_~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~"~~~~~~~~~~~~~~~~~~~';; s,(~|\r|\n|\s),,g;s.Y.\x7e.g; eval};Greet;'the world';]]></parameter> </command>

Other minor improvements

I have created a utility for users to change their own password or the password of another user. You must have been granted the “passwd” privilege to perform this operation (this is provided by jaas). All passwords are stored encrypted, currently in an xml document used for the user repository (I still have to write the LDAP functionality, it will come).

I fixed a small issue which workflow instances causing exception on the development environment when the server was stopped and re-started. Exceptions are now captured for each workflow to prevent exceptions being blocking to the server startup. Workflows that cause an exceptions would not be resumed.

What’s Coming Next?

I have to start thinking about bundling the release together. I’ve spent some time cleaning up the blog to look presentable and re-linked it to linked-in, google+, Twitter and Facebook. You never know who might be reading…

I also started a bit enthusiastically with the version numbering system. The next version will be version 1.2.2, to slow things down a notch. I have to look in the product backlog (yes, I do sprint planning) but I have some tidying up to do of the build system.
Maybe, I’d like to do something around queued tasks (no owner until someone changes the task ownership), Client Sessions and OAuth (to make it even more robust and ready for the Rest API I’ll be putting in for Version 2 [definitely a version number
changer!])

I’ll stop blabbing on now and actually organise my sprint. If I don’t see you before next year,
Have a Happy new Year!

Part of what I was developing for this sprint has been to check the permissions when attempting to change the owner of a task.

The commands

laurent@laurent-Aspire-5742:~/Projects/development/LapisServer/bin$ ./wflist.sh -u laurent -p xxxxx -host localhost -port 12345
test [32bbae32-8944-4c31-b370-60024bf533b3]
	=>	4f480ffe-21a3-4d82-8697-436c2a9fa506 [pause [laurent] [070d3c83-d658-4c22-bc1a-7c4f83660b49] => Complete]
	=>	a12ded01-ce52-4cef-81b6-274954cf8443 [Another pause [laurent] [4ff04705-9818-4637-87dc-5080bc35a50e] => Complete]
	=>	5bbf56c7-10d0-4da7-b3e7-0dc99bccd751 [verify [sarah] [30bd0552-7c1b-4fb7-97ae-34923789e9ed] => Active]
		=>	finish
		=>	start subworkflow
	=>	eab0c785-f523-4fea-afbd-c9589dc73088 [hello [laurent] [fe69d6ad-10b5-4ec8-9f1b-def5fdaa9505] => Complete]
	=>	6dbf3510-aef9-4840-9bf8-3c30bc2930dd [subWorkflow [laurent] [f9c21bfa-fa9d-46ec-950b-88b9d367d6af] => Rejected]

The user task should have been completed by the user Sarah. I then issue the command to change the owner for the task to myself, but since I don’t have the permission to change the task ownership, the engine raises an exception:

laurent@laurent-Aspire-5742:~/Projects/development/LapisServer/bin$ ./wfchown.sh -u laurent -p xxxxx -host localhost -port 12345 -w 32bbae32-8944-4c31-b370-60024bf533b3  -t 5bbf56c7-10d0-4da7-b3e7-0dc99bccd751 -o "laurent"
Exception in thread "main" java.security.AccessControlException: access denied ("fukoka.lapis.engine.security.authorisation.FukokaPermission" "wfchown")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:372)
	at java.security.AccessController.checkPermission(AccessController.java:559)
	at fukoka.lapis.engine.security.authorisation.FukokaPrivilegedAction.run(FukokaPrivilegedAction.java:38)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAsPrivileged(Subject.java:536)
	at fukoka.lapis.engine.workflow.remote.RemoteNode.setOwner(RemoteNode.java:134)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
	at sun.rmi.transport.Transport$1.run(Transport.java:177)
	at sun.rmi.transport.Transport$1.run(Transport.java:174)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194)
	at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148)
	at com.sun.proxy.$Proxy5.setOwner(Unknown Source)
	at fukoka.lapis.client.clt.WFchown.main(WFchown.java:63)

In order to succeed, I then edit the server policy file to grant the permission:

grant principal fukoka.lapis.engine.security.authentication.FukokaPrincipal "laurent" {
    permission fukoka.lapis.engine.security.authorisation.FukokaPermission "shutdown";
    permission fukoka.lapis.engine.security.authorisation.FukokaPermission "wfchown";
};

The listing now gives me the ownership of the task

Related articles

I have had a comment about the licensing model of the workflow engine. Since I have put a lot of effort into it, I would like it to be paid software and not open source. These are early days yet so I may be persuaded otherwise yet if the community clamours for it.

There are 3 valid licenses that can be obtained:

  • a 30 days evaluation license: None of the features are restricted. The license is only valid for 30 days.
  • a Developer Server license: None of the features are restricted. The license is only for non-production systems.
  • a Production Server license: None of the features are restricted.

In order to obtain a license, please contact Laurent Picquet, CEO of Fukoka Ltd. at lpicquet@gmail.com.

Licenses are server restricted, port restricted and time restricted. Any of these restrictions can be lifted. For example, a license may be issued to be valid across all the servers in an organisation, or/and across any port on a server, or/and to never expire. Licenses issued for an organisation or individual entities cannot be shared with or resold to other organisations or individual entities.

The license information comes in 2 parts. One part is a license properties file (fukoka_lapis.lic), detailing what the license covers. The other part is a signature file validating the licese properties (fukoka_lapis.sig). Tempering with any of these 2 files will invalidate the license.

The server properties file must reference these 2 files (usually in the etc sub-folder of your installation) through the following 2 properties:

  • lapis.license.licenseFile
  • lapis.license.licenseSignatureFile
The server will validate the license upon startup. Below is an example license file fukoka_lapis.lic
#Fukoka Ltd.
#Sat Nov 30 06:14:29 GMT 2013
expires=never
port=12345
servername=laurent-Aspire-5742
organisation=Fukoka Ltd.