IRI Blog Articles

Diving Deeper into Data Management

 

 

Post image for How to Run Remote Jobs from IRI Workbench

How to Run Remote Jobs from IRI Workbench

by Susan Gegner

This article covers the creation and use of remote server connections from IRI Workbench so you can run CoSort (Voracity), FACT, NextForm, FieldShield, or RowGen scripts on other computers.1

IRI Workbench is the graphical IDE client, built on Eclipse™, where you design and manage all IRI software jobs. Depending on your license agreement, your program’s executable can run on one or more Windows, Linux, or Unix servers.

We will detail these steps for setting up and running remote jobs:

  1. Set up the remote environment and connection.
  2. Create a directory on the remote server to hold your projects (optional).
  3. Create a remote project.
  4. Associate a local project with the remote project (optional).
  5. Set up the Run Configurations.
  6. Execute a job script on the remote server.

Set up the remote environment and connection

First, go through this “pre-flight” checklist. Make sure that:

  1. Your remote server is an SSH server.
  2. Your back-end IRI software product is installed and licensed on that server.
  3. You have set its “HOME” environment variable on the server. If you have CoSort or Voracity, that would be $COSORT_HOME on Unix or %COSORT_HOME% on Windows. Typically, this is the install directory for the software.
  4. The directory location of the executable is in the PATH on the remote server, though that can also be set later in the Workbench Run Configurations dialog. The executable for running CoSort or Voracity jobs installs into COSORT_HOME/bin by default, but it can be moved.

These same prerequisites apply if you are using different IRI product executables. If you use FACT, consult its manual for other configuration details.

To connect the IRI Workbench client and your remote server, you must define a New Connection in the Remote Systems view. If you do not see the Remote Systems view, click Window, Show View, Remote Systems. In the Remote Systems view, click the first icon in its toolbar.

Remote Systems

This opens the Select Remote System Type dialog. Click SSH Only. Do not select any of the other types. Now click Next.

Select Remote System Type

Next is the Remote SSH Only System Connection dialog. The Parent profile should have the name of the client computer where you are running IRI Workbench (lightning in this case).

The Hostname is the name of the remote server.  Here, the remote server is force5.iri.com and the Connection name can be force5.

The Description field gives further information about the server. Here, I entered Linux to indicate the operating system on force5. Now click Finish.

Remote SSH Only System Connection

The connection name is added to the tree in the Remote Systems view.

Now you can finish setting up the connection. Right-click on the connection name in the tree and select Connect. This dialog opens:

Enter Password

Normally, I type in the User ID and Password, then select both Save user ID and Save Password, and click OK. If you get a popup about the authenticity of the host, click Yes to continue connecting.

Expand the Sftp Files tree under force5, then expand My Home. This will show the home directory for the login we just set up for the remote system.

Create a master directory on the remote server (optional)

If you intend to build multiple projects on the remote server, it is recommended you create a directory that will hold multiple project directories below it. To do that:

  1. Highlight My Home under the Sftp Files connection that was just created.
  2. Right click and select New -> Folder.
  3. In the New Folder dialog, Connection Name and Parent folder will already be filled in.
  4. For the field New folder name, type in the directory name RemoteProjects.
  5. Click Finish. The new folder is created on the remote server and should appear in the tree under My Home in Workbench.

Remote Folder

You can then create one or more project folders in this directory on the remote server. Ideally, all project directories should be created under the RemoteProjects directory, but they can be created anywhere.

Create a remote directory and project

To create a specific remote directory and project (under our master RemoteProjects directory):

  1. Highlight RemoteProjects in the Remote Systems tree under force5.
  2. Right-click and Select New -> Folder.
  3. In the field New folder name, type in the name remote_project1.
  4. Click Finish. The new folder is created on the remote server and appears in the tree under My Home/RemoteProjects.
  5. Right-click on the newly created directory remote_project1 -> Create Remote Project.

We now have the project directory on the remote server called remote_project1, and a project has been created in the IRI Workbench Project Explorer called force5_remote_project1.

Note that if you are unable to access the files in a project, that project might be closed. To open a project, double-click on it, or right-click on the project, and select Open Project.

Associate a local project with the remote project

You might have an existing local project that you wish to deploy on a remote server. Or, if you don’t, you might still want to create a new local project and associate it with a remote project. This is because there are things you cannot do with IRI’s (new job creation) wizards in remote projects directly, like discover .DDF metadata and use Flow files. For this reason, you need to associate a local project with your remote project. Later, you can copy files into the remote project from this local project.

I give my local project the same name as my remote project, but replace remote with local in the name. On the navigation bar, select File -> New -> IRI Project. Type in the Project name force5_local_project1. Click Finish.

In the local project, you can use wizards from the Workbench navigation bar to create job scripts and metadata files. Once these have been created, they can be copied and pasted to the remote project using the available commands.

Any files can be copied, including data files. Be sure that the path for any file defined in your scripts references the path needed on the remote server.

Files, scripts, etc., that have been copied to a remote project can be opened in the editor in Workbench, and any edits will be saved into the remote project. However, flow files (.flow) cannot be opened in the remote project. The first time you edit an IRI-specific file (e.g., .scl, .fcl, .ddf, etc.), you will see this pop-up, and you must click Yes:

Configure Xtext

Next, we need to set Run Configurations for executing the remote project job(s).

Set up the Run Configurations

  1. Highlight the remote project force5_remote_project1 in the Project Explorer.
  2. Right-click on the name of a script in the project or inside the body of an open job script from that folder.
  3. Select Run As -> Run Configurations.
  4. In the panel on the left side of the Create, manage, and run configurations window that opens, right-click on IRI Job (or IRI FACT) and select New.
  5. New_configuration is added under IRI Job and is placed in the Name field in the window to the right of the panel. Replace this with a more descriptive name.  I am using the name force5_proj1_chiefs.

Now, we need to set up the tabs in this window. First, we will set up the Main tab.

The name of our remote project, force5_remote_project1, should be specified in the Project field. Or, use the Browse button to find a project. Select it, and click OK.

The Job Execution Sequence box should have the path and name for the job script that we want to run from the remote project. Select the Search/Add button for a list of files in the Project. In the Select Resource dialog, select your job script and click OK.

Select Resource

That job is now listed in the box for Job Execution Sequence. Click Apply to save these settings. This concludes the setup for the Main tab.

Run Configurations

Execute a job script on the remote server

Next, we need to populate the entries in the Remote Execution tab.

  1. Select Remote and click the Browse button for Working Directory.
  2. In the Select Working Directory dialog, select the remote server (force5) from the Connection dropdown.  
  3. In the tree below that, expand My Home. Click on the remote_project1 directory and click OK.

The Connection and Host fields are filled in as well as the Working Directory. Connection will have the Name that we gave to the remote server connection, and Host will have the host name for that server.

In most cases, the CoSort CLI Executable (sortcl*) is located in the $COSORT_HOME/bin directory. This is the case here, so I type in $COSORT_HOME/bin/sortcl. If you are using another program, like IRI FieldShield, it would be $FIELDSHIELD_HOME/bin/fieldshield. For FACT, fact* should be in $FACT_HOME; but either way, it must be in the path of the remote connection user.

If your executable is located elsewhere, then type in the path/filename for the executable, or Browse to it on the remote server. In addition, if you are sure that executable is in the connected user’s path, then you can just type in the name of the executable. Click Apply to save.

Run Configurations2

Now that the configuration is saved, you can click Run to execute the job, and/or run the job later.

Any output targets from jobs without an explicit path specified will go into in the remote project when the job is done. The Console view for force5 will show the job running and the time it took to complete.

Console View

You can re-run this script again at any time using the saved configuration force5_proj1_chiefs.

  1. Click on the drop-down arrow next to the green circled arrow on the navigation bar.
  2. Select Run Configurations.
  3. Select force5_proj1_chiefs  under IRI Job in the panel to the left.
  4. Click Run.

When you perform step 1, it is possible that the name for the job configuration will be in the drop-down list. If that is the case, then you can click on it there to run the job.

We have now demonstrated how to set up a remote project for executing remote job scripts in IRI Workbench, and how Run Configurations can be saved so you can re-run those jobs. If you need help, contact support@iri.com.

Print Friendly
  1. Local task execution was covered in this article, and remote execution of multi-table batch jobs will be covered in a later article. Execution of jobs in Hadoop is described here.

{ 0 comments… add one now }

Leave a Comment

Previous post:

Next post: