Set Up and Develop Thought Processes

This page describes how to set up the Thought Process Engine (TPE) service (knowns as THOP service) and how to develop THOPs using the THOPBuilder plug-in. To gain an overall understanding of Thought Processes and view examples, go to Thought Processes Overview and Examples.

Set up the Thought Process Engine Service

THOPs 1.0

To develop THOPs 1.0, you must install THOPBuilder 1.0 in IntelliJ and point it to the standard ws service in SMB. Follow the instructions under Develop THOPs using THOPBuilder.

THOPs 2.0

To develop THOPs 2.0 (for asynchronous and potentially forver-running THOPs), you or your system administrator must install saffron-ws. It has a REST API to store and run THOPs. You must also install IntelliJ^® and Java 8. Follow the instructions under Develop THOPs using THOPBuilder.

Install Saffron-WebServices

Prerequisites

In order for the TPE service to run, a standard Java 8 installation is required.

To install Java 8 (OpenSDK) on CentOS, do the following:

$ sudo yum install java-1.8.0-openjdk-headless.x86_64

To verify if Java 8 has been correctly installed, run:

$ java -version

Verify that the version is 1.8.0_*

If a different java version is installed, run the alternatives command to select the correct version:

$ sudo alternatives --config java

(alternatives is part of the chkconfig package)

If you are installing Oracle 8 instead, follow the instructions here: http://tecadmin.net/install-java-8-on-centos-rhel-and-fedora/#

Note: If you do not install this on a system where Saffron is already installed, create a 'saffron' user:

$ adduser saffron

Installation

Download the Saffron WS RPM: Saffron-WS--<version>-<build>.noarch.rpm

$ sudo service saffron-ws stop

Note: The stop command will fail if you do not have an earlier version of Saffron-WS installed.

$ sudo yum install Saffron-WS--<version>-<build>.noarch.rpm
$ sudo service saffron-ws start

The service is available by default on port 8888.

To check if the initial installation was successful, use `curl`:

$ curl 'http://localhost:8888/ws/rest/proc'
{"stored_procedures" : [ ]
}

Note: If you have already configured user authentication as outlined in the Security section below, use curl -u user:pwd 'http://localhost:8888/ws/rest/proc'. The user must have the ROLE_ADMIN or the ROLE_API_USER role.

Configuration 

Configuration files for the TPE are available in /etc/saffron-ws.d

The standard configuration files can be found in the defaults sub-directory.

To make changes to a configuration file, copy the appropriate file from /etc/saffron-ws.d/defaults to /etc/saffron-ws.d and edit it there.

Check the configuration files for documentation about all of its settings.

It is recommended to adjust the following settings to tune the performance and security of the TPE service:

server.properties:

# Maximum number of concurrent HTTP connections to SMB instances - per host

max.connections.per.host=20

# Set the authentication/authorization provider.

# Possible values:

# none - no authentication for all REST resources

# smb10 - authenticate using the login services of a Saffron v10 instance. Needs a valid smbURL configured (see below)
security=none

# Optional URL to the SMB system used for authentication purposes.

# Important: URL should end in a slash, default value: http://localhost:8080/ws/

#

# NOTE:

# This setting is only used if security=smb10 (see above)

smbURL=

You can also change the IP interfaces the service binds to and port number if there are port number conflicts.

Lastly, if THOPs need to make connections to HTTPS services requiring private certificates, change the client.keystore.file setting (this is often not required as THOPs are typically only making calls to a local SaffronMemoryBase instance using HTTP).

cache.properties:

# Settings for the query results cache.

# Caching is based on ETags received from SMB instances
# Size of the cache in bytes, k, kb, kib, m, mb, mib, g, gb, gib
# This is the amount of memory reserved for storing query results.
# The cache might allocated more than the configured amount if it can fit one more entry before filling up.
# Cache entries are removed using an LRU algorithm

cache.size = 10mb

If the TPE installation is expected to see large amount of queries or large query results from SaffronMemoryBase, increasing this value is recommended. The cache is an in-memory, compressing LRU cache and its contents are lost on service restart.

Note: For the config changes to take effect, issue:

$ service saffron-ws restart

Security

By default, a TPE installation is not secured and anyone can upload and run THOPs.

To secure TPE and tie its user management to an existing SMB system, change the security and smbURL setting in server.properties to point to an existing SMB system.

Sample configuration:

# Maximum number of concurrent HTTP connections to SMB instances - per host
max.connections.per.host=20

# Set the authentication/authorization provider.
# Possible values:
# none - no authentication for all REST resources
# smb10 - authenticate using the login services of a Saffron v10 instance. Needs a valid smbURL configured (see below)
security=smb10

# Optional URL to the SMB system used for authentication purposes.
# Important: URL should end in a slash, default value: http://localhost:8080/ws/
#
# NOTE:
# This setting is only used if security=smb10 (see above)
smbURL=http://localhost:8080/ws/

This will secure the `/ws/rest/proc` paths using HTTP Basic Authentication.  

Note: In order for this config change to take effect, issue:

$ service saffron-ws restart

Exposing TPE service when part of a SaffronMemoryBase cluster (HTTP/S support)

The TPE service can be used stand-alone or as part of an SaffronMemoryBase cluster. 

If part of an SaffronMemoryBase cluster (typically installed on the SaffronMemoryBase head node) the service can be exposed via HTTP/S using the SaffronMemoryBase cluster Apache proxy.

To do so, add proxy pass rules to the <VirtualHost> sections (one for port 80 and one for port 443) in /etc/httpd/conf.d/saffron_https_proxy.conf

ProxyPass /thop http://127.0.0.1:8888/ws
ProxyPassReverse /thop http://127.0.0.1:8888/ws

Refer to the Apache documentation for more information about these settings.

Maintenance

Log files for this service can be found in /var/log/saffron-ws

Control over the service is accomplished through regular unix tools like service and chkconfig.

Thought Processes uploaded via the REST API are stored in/var/tmp/thops (subject to change in future versions of this package) 

Backup/Restore

The THOPs stored in the current version of the TPE service are considered ephemeral (hence the /var/tmp/thops location).

Users of the TPE service are expected to re-upload missing THOPs during bootstrapping.

If a snapshot of the state of the TPE service is required, use any available tool to store the contents of /var/tmp/thops.

To guarantee consistency of the snapshot, stop the service first.

Troubleshooting

If service saffron-ws start fails, check the log files for a more detailed problem description.

If error contains this:
com/saffrontech/ws/vertx/Server : Unsupported major.minor version 52

make sure that the saffron user is using the correct version of Java 8.

Compare paths from which Java is loaded of root & saffron user:

$ which java
$ su saffron
$ java -version
$ which java

Adjust ~saffron/.bashrc if necessary.

Use NPM as a package manager with THOPs 2.0

THOPs 2.0 supports access to libraries that can be used from within THOPs. See Use NPM with THOPs.

Develop THOPs using THOPbuilder

Prerequisites

The following must be installed before you install THOPBuilder:

• IntelliJ IDE
  
  Saffron offers an Intellij IDE plug-in in order to develop, test, and run Thought Processes (THOPs). This plug-in works for all editions of IntelliJ IDEA. The Community Edition is available for free.

• Java 8 (THOPs 2.0 only)
  
  You must install Java 8 in order to use THOPS 2.0.

Install the THOP Builder plug-in

1. Download one of the attached .zip files below (do not unzip this file):
   
   THOPBuilder-2.0.2.zip (Make sure Java 8 is already installed.)
   DownloadDownload 
   THOPBuilder-1.3.4.zip (Java 8 is not necessary.)
   DownloadDownload  
2. Add the THOP builder as a plugin to IntelliJ:
   
   Open Preferences / Plugins / Install plugin from disk

   Select the downloaded THOPBuilder zip file from Step 1. (Do not unzip the file.)

3. Restart IntelliJ.
4. Re-open the Preferences / Plugins window and confirm that it contains the THOP Builder plug-in.

Set up a connection to a SaffronMemoryBase

Connections to SaffronMemoryBase are stored per IDEA project.

1. To create a new static web project, click File / New / Project.
   
   You do not need to enter a Project SDK. Simply create an empty project.
2. Open Tools and select Connect to SMB in the Project you just created.
3. Enter the connection data, as shown in the example below.

   If you are not running SaffronMemoryBase on your local machine, point it to the URL used to log in to the Saffron Admin tool. Use /ws instead of /admin at the end of the URL.

   While the connection is being made with Username and Password, the Access Key is also required for THOPs to make Queries to SaffronMemoryBase.

   
    
4. Click Test to confirm a successful connection.
5. Click OK and follow the instructions to download THOPs to test if the connection works as expected. Make certain your user has the ROLE_API_USER permissions.
   
   Note: You must have SaffronMemoryBase 10.2 for this to work properly. If you are running 10.1, the test result might be 'Can't log in' even if you provide the correct credentials.
6. Click OK and continue.

THOP Builder Features

Download THOPs to Intellij

1. Create a directory in the new project you just created (the Download THOPs action only works on directories).
2. Right-click that directory and choose Download THOPs.
   
   The currently-installed Thought Processes for the configured SaffronMemoryBase instance are downloaded as files and saved to the directory.

Upload THOPs to SaffronMemoryBase

1. Right-click one or more of the existing THOPs or create a new .js.
2. Choose Upload THOP in the context menu.
   
   This uploads the current contents of the selected file(s) and overwrites any existing Thought Processes by the same name. The name is taken from the filename without extension.

Delete THOPs from SaffronMemoryBase

1. Select one or more THOPs you want to delete.
2. Right-click the selected THOPs and select Delete THOPs from the context menu. This remotely removes the THOPs by the same name.
   
   Note: Local files are not deleted. If you delete a THOP by accident, you can re-upload it.

Run THOPs

Right-click a THOP (for example, HelloWorld.js) and perform one of the following:

• Run HelloWorld ... Right-click HelloWorld and select Run HelloWorld. This creates a default run configuration and then uploads and runs the Hello World THOP. The console window shows the results of running the THOP.
• Debug HelloWorld ... Right-click HelloWorld and select Debug HelloWorld. This creates a default run configuration and uploads and runs the Hello World THOP with debug information enabled. The console window shows the result of the THOP as well as any debug information collected using debug(...) calls within the THOPs code. Note: Debug output in THOPs is displayed only if you use the Debug action to run the THOP.

Edit the Run Configuration

The Run/Debug Configurations window captures parameters and other available options to run a THOP.

To open, select Edit Configurations for a THOP.

The Run/Debug Configurations window opens:

Parameters can be specified one per line using the format: parameter=value. Values are automatically URL-encoded and can be specified with blanks and other reserved characters.

Upload THOP before running is selected by default. When the THOP is run, the file specified in Local THOP is uploaded first.

Defaults: Set the default SMB (and default values) to which the THOP communicates. You can add arbitrary JSON objects here (THOP Builder will only support simple properties though). They can be accessed inside the THOP using the new defaults variable. If you upload via POST/PUT, add JSON object defaults to your existing JSON payload.

Together with the main toolbar's Run feature, this process leads to a seamless coding and test cycle without having to upload the THOP manually. Note that you can also save a Run configuration and create multiple run configurations for the same THOP for testing.

Troubleshoot Issues

No THOPs are downloaded: Check the Event Log in Intellij. Check your connection settings in Tools / Connect SMB...

Test button doesn't work, i.e. an error message is displayed: If the error is Can't log in, you might be connecting to an older version of SMB. Open your browser, enter the SMB's hostname and port number followed by /ws/rest/spaces. Then, log in with your credentials.

I still can't log in: Verify that your user has the correct roles. Prior to 10.2, only the ROLE_API_USER is required. After 10.2, both the ROLE_API_USER and ROLE_DEVELOPER are required.

My user is an admin and I can't log in: Prior to 10.2 admin users need to be given the ROLE_API_USER role. Admin users don't require special roles when using SMB 10.2 or later.

Advanced Topics

Use NPM with THOPs

THOPs 2.0 supports access to libraries that can be used from within THOPs. NPM provides JavaScript libraries that can be used by THOPs 2.0. Install NPM as the package manager (https://www.npmjs.com/package/npm).

To use packages maintained by NPM in THOPs 2.0, do the following:

1. In your shell/IDE, set the environment variable NODE_PATH to ./node_modules (or wherever you want npm modules to exist).
2. Start Saffron-WS.

To install a package, go to the home directory of Saffron-WS (adjust if you changed NODE_PATH above) and run:

npm install <package>

This installs the named package in ./node_modules.

To use the package, use the require function in your THOP (more specific instructions are usually in the package documentation).

Example of using an NPM package

cd where/my/ws-fat.jar/file/is
export NODE_PATH=./node_modules
npm install n-gram

THOP code:

function run(p) {
    var nGram = require('n-gram');
    return nGram.bigram('Saffron Rocks');
 }

Note: There is currently no simple flag that tells you if a particular package runs on Nashorn or not. Check the dependencies of the package (listed on npmjs.com). Preferably there are none. Also, if the documentation mentions that the code runs in the browser, there's a good chance it will run in Nashorn.