Page Comparison

Beginning with yuuvis yuuvis® RAD version 9.0, the Extraction Service extraction-service can be extended by plugins.
This is based on the framework PF4J: https://pf4j.org/
Inside the extractionservice the plugins are loaded via PF4J-Spring:
https://github.com/pf4j/pf4j-spring
This means that a plugin must comply with this interface. In particular, this is a constructor with a PluginWrapper as a parameterplug-ins.

Table of Contents

minLevel	1
maxLevel	7

Including an Existing Plug-in in the Extraction-Service

The deployment of an existing plug-in is explained by setting up an example plug-in for processing PGN files.

Configuring the Extraction-Service

By default, the use of plug-ins is disabled. In the extraction-prod.yml file, these settings can be adjusted:

Code Block

language	javayaml

	public PgnFilePlugin(PluginWrapper wrapper) {
plugins:
  enabled: true
       super(wrapper);
    }

Necessary steps (not chronological)

It follows the description of what to set up and configure for creating a new plugin, deploying, and checking it.

1. Include the library provided by Optimal Systems for the definition of the plugin interface

The library "extractionservice-plugin-interface" is available as a jar file and must be added to the project. This defines the interface for the extraction service and also brings the plugin capability of PF4J.

View file

name	extractionservice-plugin-interface-9.0.2.jar

It is important that the dependency of the plugin on the interface library is defined as "provided" (in gradle "compileOnly"):

Code Block

language	xml

<dependency>
			<groupId>com.os.enaio.services.extraction</groupId>
			<artifactId>extractionservice-plugin-interface</artifactId>
			<version>9.0.2</version>
			<scope>provided</scope>
		</dependency>

2. Implement the plugin

As an example of a plugin the library "extractionservice-plugin-demo" is also available as a jar-file:

View file

name	extractionservice-plugin-demo-9.0.0-all.jar

This Example operates on PGN-Files: https://en.wikipedia.org/wiki/Portable_Game_Notation

View file

name	chess_PGN.pgn

Plugin example code:

Code Block

language	java

package com.os.services.extraction.plugin.demo;

import com.os.services.extraction.plugin.ExtractionDriver;
import com.os.services.extraction.plugin.ExtractionInfo;
import org.apache.commons.io.IOUtils;
import org.pf4j.Extension;
import org.pf4j.Plugin;
import org.pf4j.PluginWrapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.FileInputStream;
import java.util.HashMap;
import java.util.Map;

public class PgnFilePlugin extends Plugin {

    /**
     * Constructor to be used by plugin manager for plugin instantiation.
     * Your plugins have to provide constructor with this exact signature to
     * be successfully loaded by manager.
     *
     * @param wrapper
     */
    public PgnFilePlugin(PluginWrapper wrapper) {
        super(wrapper);
    }
    private final static Logger LOGGER = LoggerFactory.getLogger(PgnFilePlugin.class);

    @Override
    public void start() {
        LOGGER.info("Start");
    }

    @Override
    public void stop() {
        LOGGER.info("Stop");
    }

    @Extension
    public static class PgnFileProcessor implements ExtractionDriver {
        private final static Logger LOGGER = LoggerFactory.getLogger(PgnFileProcessor.class);

        private static final String PREFIX = "PGNFILE";

        @Override
        public ExtractionInfo info() {
            return null;
        }

        @Override
        public String getExtractionDriverName() {
            return "PgnFile";
        }

        @Override
        public String getExtractionNamespace() {
            return PREFIX;
        }

        @Override
        public boolean isApplicable (File file) {

            if (file.getName().endsWith(".pgn")) {
                return true;
            }
            return false;
        }

        @Override
        public Map<String, String> extractData( File file ) {

            Map<String, String> result = new HashMap<>();
            try {
                String str = IOUtils.toString(new FileInputStream(file), "UTF-8");

                String[] tokens = str.split("]");
                if (tokens != null) {
                    for (String token : tokens) {
                        token = token.trim();
                        if (token.startsWith("[")) {
         directory: '../../data/plugins'

The jar file of the plug-in must then be stored in the specified directory:

View file

name	extractionservice-plugin-demo-9.0.0-all.jar

The extraction-service must be restarted to enable the plug-in.
Any number of plug-in files can be stored in this directory.

Checking the Logging of the Plug-in

As soons as the extraction-service has been restarted, available plug-ins are detected. Information on these plug-ins can be found in the log.

If the plugins.enabled switch is not set to true, the following will be protocolled in the log:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor          : plugins not enabled

If the plug-ins are enabled, but not stored in the directory, the log will look as follows:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor          : plugins enabled: ../../data/plugins
... c.o.s.e.routing.PluginProcessor          : plugins found: 0

If plug-ins are found, they will be listed with their names and namespaces:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor          : plugins enabled: ./../data/plugins
... c.o.s.e.routing.PluginProcessor          : plugins found: 2
... c.o.s.e.routing.PluginProcessor          :   plugin 'PgnFile': PGNFILE
... c.o.s.e.routing.PluginProcessor          :   plugin 'CAD-File': CAD

If plug-ins are active for a specific file, the number of each extracted plug-in will be in the log:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor          : extracted by plugin 'PgnFile': 7

Result of the Extraction

View file

name	chess_PGN.pgn

The result of the extraction from this PGN file will look like this:

Code Block

language	text

 "OS:FileName": "schach_PGN.pgn",
 "OS:FileSize": "740",
 "OS:ModifyDate": "2023-02-10T09:42:24+01:00",
 "OS:CreateDate": "2023-02-10T09:42:24+01:00",
 "OS:FileType": "TXT",
 "OS:MimeType": "text/plain",
 "ExifTool:ExifToolVersion": "12.52",
 "System:FileName": "schach_PGN.pgn",
 "System:Directory": "C:/Temp/Local/Temp/20230210.094224.115-1/389a8d00-25f8-4f84-9d08-238f0e81594c",
 "System:FileSize": "740",
 "System:FileModifyDate": "2023:02:10 09:42:24+01:00",
 "System:FileAccessDate": "2023:02:10 09:42:24+01:00",
 "System:FileCreateDate": "2023:02:10 09:42:24+01:00",
 "System:FilePermissions": "100666",
 "File:FileType": "TXT",
 "File:FileTypeExtension": "TXT",
 "File:MIMEType": "text/plain",
 "File:MIMEEncoding": "us-ascii",
 "File:Newlines": "\n",
 "File:LineCount": "16",
 "File:WordCount": "158",
 "PGNFILE:Site": "Belgrade, Serbia JUG",
 "PGNFILE:White": "Fischer, Robert J.",
 "PGNFILE:Event": "F/S Return Match",
 "PGNFILE:Round": "29",
 "PGNFILE:Black": "Spassky, Boris V.",
 "PGNFILE:Date": "1992-11-04T12:00:00+01:00",
 "PGNFILE:Result": "1/2-1/2",

Creating a Plug-in

This section describes how to set up and configure a new plug-in including its deployment and check.

The plug-in capability of the extraction-service is based on the PF4J framework.
The extraction-service loads plug-ins via PF4J-Spring.

The plug-in must comply with the interface defined by PF4J-Spring. In particular, this is a constructor with a PluginWrapper as a parameter:

Code Block

language	java

	public PgnFilePlugin(PluginWrapper wrapper) {
        super(wrapper);
    }

Including the Library for the Definition of the Plug-in Interface

OPTIMAL SYSTEMS provides a library for the defintion of the plug-in interface. The extractionservice-plugin-interface.jar file is available for download below and must be added to the project. It defines the interface for the extraction-service and also brings the plug-in capability of PF4J.

View file

name	extractionservice-plugin-interface-9.0.2.jar

It is important that the dependency of the plug-in to the interface library is defined as provided (in gradle compileOnly):

Code Block

language	xml

<dependency>
			<groupId>com.os.enaio.services.extraction</groupId>
			<artifactId>extractionservice-plugin-interface</artifactId>
			<version>9.0.2</version>
			<scope>provided</scope>
		</dependency>

Implementing the Plug-in

As an example of a plug-in, the extractionservice-plugin-demo.jar file is also avaible for download:

View file

name	extractionservice-plugin-demo-9.0.0-all.jar

The following example operates on PGN files.

Plug-in example code:

Code Block

language	java

package com.os.services.extraction.plugin.demo;

import com.os.services.extraction.plugin.ExtractionDriver;
import com.os.services.extraction.plugin.ExtractionInfo;
import org.apache.commons.io.IOUtils;
import org.pf4j.Extension;
import org.pf4j.Plugin;
import org.pf4j.PluginWrapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.FileInputStream;
import java.util.HashMap;
import java.util.Map;

public class PgnFilePlugin extends Plugin {

    /**
     * Constructor to be used by plugin manager for plugin instantiation.
     * Your plugins have to provide constructor with this exact signature to
     * be successfully loaded by manager.
     *
     * @param wrapper
     */
    public PgnFilePlugin(PluginWrapper wrapper) {
        super(wrapper);
    }
    private final static Logger LOGGER = LoggerFactory.getLogger(PgnFilePlugin.class);

    @Override
    public void start() {
        LOGGER.info("Start");
    }

    @Override
    public void stop() {
        LOGGER.info("Stop");
    }

    @Extension
    public static class PgnFileProcessor implements ExtractionDriver {
        private final static Logger LOGGER = LoggerFactory.getLogger(PgnFileProcessor.class);

        private static final String PREFIX = "PGNFILE";

 String[] tag = token.substring(1).split("\"");    @Override
        public ExtractionInfo info() {
            if (tag.length >= 2) {return null;
        }

        @Override
        public String getExtractionDriverName() {
    //do not add prefix to key   return "PgnFile";
        }

        @Override
         //it will be set by extractionservicepublic String getExtractionNamespace() {
            return PREFIX;
        }

        @Override
        Stringpublic keyboolean = tag[0].trim();isApplicable (File file) {

            if (file.getName().endsWith(".pgn")) {
                Stringreturn valuetrue;
= tag[1].trim();           }
            return false;
        if (key.endsWith("Date")) {}

        @Override
        public Map<String, String> extractData( File file )  {

          value = value.replaceAll("\\?{2}", "01")
    Map<String, String> result = new HashMap<>();
            try {
                String str =        .replaceAll("\\."IOUtils.toString(new FileInputStream(file), "\\UTF-8") + "T12:00:00+01:00";

                String[] tokens = str.split("]");
           }     if (tokens != null) {
                    for (String token  result.put(key, value);
: tokens) {
                        token   }
   = token.trim();
                        if (tag.length == 1token.startsWith("[")) {
                            String[] tag =  //do not add prefix to keytoken.substring(1).split("\"");
                            if (tag.length >= 2) {
     //it will be set by extractionservice                      //do not add prefix to key
     String key = tag[0].trim();                        //it will be set by extractionservice
    result.put(key, "");                           String key }
= tag[0].trim();
                        }        String value = tag[1].trim();

        }                 }             } catch (Exception eif (key.endsWith("Date")) {

               LOGGER.error(e.getMessage());             }        value = value.replaceAll("\\?{2}", "01")
   return result;                                  }      }
}

3. Build the jar-file

If the plugin is packaged in a jar file, there are important points to note:
The following attributes must be specified in the MANIFEST.MF file (in the example plugin):

Code Block

language	text

Plugin-Class: com.os.services.extraction.plugin.demo.PgnFilePlugin
Plugin-Dependencies:
Plugin-Id: PgnFileProcessor-Plugin
Plugin-Provider: os
Plugin-Version: 9.0.0

To have these values automatically inserted into the jar file, the "maven-assembly-plugin" can be configured accordingly:

Code Block

language	xml

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-assembly-plugin</artifactId>
				<version>3.1.0</version>
				<configuration>
					<descriptorRefs>
						<descriptorRef>jar-with-dependencies</descriptorRef>
					</descriptorRefs>
					<finalName>${project.artifactId}-${project.version}-all</finalName>
					<appendAssemblyId>false</appendAssemblyId>
					<attach>false</attach>
					<archive>
						<manifest>
							<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
							<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
						</manifest>
						<manifestEntries>
							<Plugin-Id>${plugin.id}</Plugin-Id>
							<Plugin-Version>${plugin.version}</Plugin-Version>
							<Plugin-Provider>${plugin.provider}</Plugin-Provider>
							<Plugin-Class>${plugin.class}</Plugin-Class>
							<Plugin-Dependencies>${plugin.dependencies}</Plugin-Dependencies>
						</manifestEntries>
					</archive>
				</configuration>
				<executions>
					<execution>
						<id>make-assembly</id>
						<phase>package</phase>
						<goals>
							<goal>single</goal>
						</goals>
					</execution>
				</executions>
			</plugin>

The jar file must contain all required libraries (except extractionservice-plugin-interface and thus also pf4j).
The MANIFEST.MF file must have the appropriate entries.
The extensions.idx file must be included. This is automatically generated by PF4J during build.

This generated jar file can now be placed in the appropriately configured directory of the Extractionservice.4

4. Configure the extraction service

By default, the use of plugins is disabled. In the extraction-prod.yml file, these settings can be adjusted:

Code Block

language	yaml

plugins:
  enabled: true
  directory: "../../data/plugins"

The jar file of the plugin must then be stored in the specified directory and is read when the Extraction Service is restarted.
Any number of plugin files can be stored in this directory.

5. Check the logging of the plugin

When the extraction service is restarted, available plug-ins are read in. Information about this is output in the log.

If the "plugins.enabled" switch has not been set to "true", this will be indicated in the log:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor .replaceAll("\\.", "\\-") + "T12:00:00+01:00";
                                }

                                result.put(key, value);
                            }
                            if (tag.length == 1) {
                                //do not add prefix to key
                                //it will be set by extractionservice
                                String key = tag[0].trim();

                         : plugins not enabled

If the plugins have been activated but no plugins have been put in the directory:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor     result.put(key, "");
        : plugins enabled: ../../data/plugins ... c.o.s.e.routing.PluginProcessor          : plugins found: 0

If plugins are found, they are listed with their names and "namespaces":

Code Block

language	text

... c.o.s.e.routing.PluginProcessor    }

              : plugins enabled: ./../data/plugins ... c.o.s.e.routing.PluginProcessor     }
    : plugins found: 2 ... c.o.s.e.routing.PluginProcessor          : }
 plugin 'PgnFile': PGNFILE ... c.o.s.e.routing.PluginProcessor          : }
 plugin 'CAD-File': CAD

When the plugins become active for a file, the number of each extracted is in the log:

Code Block

language	text

... c.o.s.e.routing.PluginProcessor               } catch (Exception e) {
    : extracted by plugin 'PgnFile': 7

6. Result of the extraction

The result of the extraction from the PGN file will look like this:

Code Block

language	text

 "OS:FileName": "schach_PGN.pgn",
 "OS:FileSize": "740",
 "OS:ModifyDate": "2023-02-10T09:42:24+01:00",
 "OS:CreateDate": "2023-02-10T09:42:24+01:00",
 "OS:FileType": "TXT",
 "OS:MimeType": "text/plain",
 "ExifTool:ExifToolVersion": "12.52",
 "System:FileName": "schach_PGN.pgn",
 "System:Directory": "C:/Temp/Local/Temp/20230210.094224.115-1/389a8d00-25f8-4f84-9d08-238f0e81594c",
 "System:FileSize": "740",
 "System:FileModifyDate": "2023:02:10 09:42:24+01:00",
 "System:FileAccessDate": "2023:02:10 09:42:24+01:00",
 "System:FileCreateDate": "2023:02:10 09:42:24+01:00",
 "System:FilePermissions": "100666",
 "File:FileType": "TXT",
 "File:FileTypeExtension": "TXT",
 "File:MIMEType": "text/plain",
 "File:MIMEEncoding": "us-ascii",
 "File:Newlines": "\n",
 "File:LineCount": "16",
 "File:WordCount": "158",
 "PGNFILE:Site": "Belgrade, Serbia JUG",
 "PGNFILE:White": "Fischer, Robert J.",
 "PGNFILE:Event": "F/S Return Match",
 "PGNFILE:Round": "29",
 "PGNFILE:Black": "Spassky, Boris V.",
 "PGNFILE:Date": "1992-11-04T12:00:00+01:00",
 "PGNFILE:Result": "1/2-1/2",        LOGGER.error(e.getMessage());
            }

            return result;
        }

    }
}

Building the `jar` File

If the plug-in is packaged in a jar file, there are important points to note:
The following attributes must be specified in the MANIFEST.MF file (in the example plug-in):

Code Block

language	text

Plugin-Class: com.os.services.extraction.plugin.demo.PgnFilePlugin
Plugin-Dependencies:
Plugin-Id: PgnFileProcessor-Plugin
Plugin-Provider: os
Plugin-Version: 9.0.0

for these values to be automatically inserted into the jar file, the maven-assembly-plugin can be configured accordingly:

Code Block

language	xml

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-assembly-plugin</artifactId>
				<version>3.1.0</version>
				<configuration>
					<descriptorRefs>
						<descriptorRef>jar-with-dependencies</descriptorRef>
					</descriptorRefs>
					<finalName>${project.artifactId}-${project.version}-all</finalName>
					<appendAssemblyId>false</appendAssemblyId>
					<attach>false</attach>
					<archive>
						<manifest>
							<addDefaultImplementationEntries>true</addDefaultImplementationEntries>
							<addDefaultSpecificationEntries>true</addDefaultSpecificationEntries>
						</manifest>
						<manifestEntries>
							<Plugin-Id>${plugin.id}</Plugin-Id>
							<Plugin-Version>${plugin.version}</Plugin-Version>
							<Plugin-Provider>${plugin.provider}</Plugin-Provider>
							<Plugin-Class>${plugin.class}</Plugin-Class>
							<Plugin-Dependencies>${plugin.dependencies}</Plugin-Dependencies>
						</manifestEntries>
					</archive>
				</configuration>
				<executions>
					<execution>
						<id>make-assembly</id>
						<phase>package</phase>
						<goals>
							<goal>single</goal>
						</goals>
					</execution>
				</executions>
			</plugin>

The jar file must contain all required libraries (except extractionservice-plugin-interface and thus also pf4j).
The MANIFEST.MF file must have the appropriate entries.
The extensions.idx file must be included. It is automatically generated by PF4J during build.

This generated jar file can now be placed in the appropriately configured directory of the extraction-service.

Checking the Plug-in

To check the new plugin, the extraction-service must be configured as described in Including an existing plugin in the extractionservice.

Versions Compared

Old Version 5

New Version Current

Key

Including an Existing Plug-in in the Extraction-Service

Configuring the Extraction-Service

Necessary steps (not chronological)

1. Include the library provided by Optimal Systems for the definition of the plugin interface

2. Implement the plugin

Checking the Logging of the Plug-in

Result of the Extraction

Creating a Plug-in

Including the Library for the Definition of the Plug-in Interface

Implementing the Plug-in

3. Build the jar-file

4. Configure the extraction service

5. Check the logging of the plugin

6. Result of the extraction

Building the `jar` File

Checking the Plug-in

Page Comparison

Versions Compared

Old Version 5

New Version Current

Key

Including an Existing Plug-in in the Extraction-Service

Configuring the Extraction-Service

Necessary steps (not chronological)

1. Include the library provided by Optimal Systems for the definition of the plugin interface

2. Implement the plugin

Checking the Logging of the Plug-in

Result of the Extraction

Creating a Plug-in

Including the Library for the Definition of the Plug-in Interface

Implementing the Plug-in

3. Build the jar-file

4. Configure the extraction service

5. Check the logging of the plugin

6. Result of the extraction

Building the jar File

Checking the Plug-in

Building the `jar` File