...

...

Resources & Remarks

Version 2019 Winter

Modification History

...

...

Introduction

Compound documents are in principle concatenations of the binary coding of several individual documents, which can be translated back into the individual documents by means of the intervals (ranges) in which the binary content of individual documents can be found. In this tutorial, a short Java application will be created that will form a simple example compound document and demonstrate the import of these documents. 

Requirements

To work through this tutorial, the following is required:

• Set-up yuuvis® API system (see Installation Guide)

• Configured user with appropriate permissions ("clouduser:secret") on tenant "default"
• Simple Maven project

Maven Configuration

Our Java client will submit its requests to the Core API using OkHttp 3.12 by Square, Inc. To build up the metadata of a compound document and evaluate the responses of the Core API, it also requires a JSON library, with org.json selected in this tutorial. To work with these libraries, the following block must be added to the Maven dependencies in the pom.xml of the project:

<dependencies>
    <dependency>
        <groupId>com.squareup.okhttp3</groupId>
        <artifactId>okhttp</artifactId>
        <version>3.12.0</version>
    </dependency>
        <dependency>
        <groupId>org.json</groupId>
               <artifactId>json</artifactId>
        <version>20180813</version>
        </dependency>
</dependencies>

Client Configuration

The basis for accessing the Core API is an OkHttp3 client that can issue HTTP requests against reachable URLs. Additionally, we need to define some variables that our OkHttp3 client will use to reach and authenticate at the Core API.

String baseUrl = "http://127.0.0.1"; //baseUrl of gateway: "http://<host>:<port>"
String username = "clouduser";
String userpassword = "secret";
String tenant = "default";
String auth = java.util.Base64.getEncoder().encodeToString((username + ":" + userpassword).getBytes());
  
OkHttpClient.Builder builder = new OkHttpClient.Builder();
OkHttpClient client = builder.build();

For more information on setting up the OkHttp3 client with cookie handling, see this tutorial on logging into the Core API.

Structure of a Compound Document

Compound documents, like all documents in the yuuvis® API system, consist of content and the associated metadata. The content of a compound document consists of the binary content of the documents contained in the compound document, which we will call subdocuments for the sake of simplicity. To ease the retrieval of the individual subdocuments, an additional set of metadata for each subdocument is imported, each with reference to the specific intervals (ranges) of byte indices denoting the location of the content of the subdocument in its respective compound document. In order to learn how to construct a compound document, we must therefore take a look at both the structure of the binary content and the structure of the metadata.

Creating the Content

To compose the content of the compound document, we first need the binary content of each subdocument. FileUtils.readFileToByteArray (File file) allows you to convert the contents of a file into a ByteArray (transforming it into binary code) that can then be written into our compound document file, or rather its FileOutputStream representation. During that process it's important to save the intervals (ranges) denoting the position of the written ByteArray(s) within the compound document file for each subdocuments' content. To do this, set an auxiliary variable offset to 0 at the beginning of the compound document creation process. For each subdocument added to the compound document, increase offset by the length of the subdocuments' content ByteArray, saving a tuple of the previous and new offset value as the range for the subdocument. That tuple will then be written into the ContentStream object of the subdocuments' metadata.

In the following Java code, the content of a compound document is assembled from three simple text files.

...

...

Introduction

Compound documents in yuuvis® Momentum are document objects with a byte array as binary content file. In the byte array, it is possible to concatenate the binary coding of multiple individual files. Sub-documents can be defined such that they refer exactly to the ranges within the byte array where the individual original files are located. A retrieval request for the content file of such sub-documents then returns exactly the binary coding of the individual original files. But also any other range or any combination of ranges can be referenced as content of sub-documents. A retrieval request for the content file of such sub-documents then returns the concatenation of the specified ranges.

In this tutorial we provide an example szenario for the handling of compound documents and sub-documents.

Requirements

To work through this tutorial, the following is required:

• Set-up yuuvis® API system (see Installation Guide)

• Configured user with appropriate permissions (in the example: clouduser:secret on tenant default)
• Simple Maven project

Maven Configuration

Our Java client will submit its requests to the Core API using OkHttp 3.12 by Square, Inc. To build up the metadata of a compound document and evaluate the responses of the Core API, it also requires a JSON library, with org.json selected in this tutorial. To work with these libraries, the following block must be added to the Maven dependencies in the pom.xml of the project:

<dependencies>
    <dependency>
        <groupId>com.squareup.okhttp3</groupId>
        <artifactId>okhttp</artifactId>
        <version>3.12.0</version>
    </dependency>
        <dependency>
        <groupId>org.json</groupId>
               <artifactId>json</artifactId>
        <version>20180813</version>
        </dependency>
</dependencies>

Client Configuration

The basis for accessing the Core API is an OkHttp3 client that can issue HTTP requests against reachable URLs. Additionally, we need to define some variables that our OkHttp3 client will use to reach and authenticate at the Core API.

String baseUrl = "http://127.0.0.1"; //baseUrl of gateway: "http://<host>:<port>"
String username = "clouduser";
String userpassword = "secret";
String tenant = "default";
String auth = java.util.Base64.getEncoder().encodeToString((username + ":" + userpassword).getBytes());
  
CookieJar cookieJar = new JavaNetCookieJar(new CookieManager(null, CookiePolicy.ACCEPT_ALL));
OkHttpClient client = new OkHttpClient.Builder().cookieJar(cookieJar).build();

For more information on setting up the OkHttp3 client with cookie handling, see this tutorial on logging into the Core API.

Binary Content of the Compound Document

The binary content of a compound document must be a byte array. All individual files you want to concatenate have to be converted into this format. In the example code block below, FileUtils.readFileToByteArray (File file) is used to convert the contents of a file into a ByteArray (transforming it into binary code). If you want to reference the content of the individual original files lateron, you need to know their length within the byte array to determine their ranges.

byte[] document1BA = FileUtils.readFileToByteArray(new File("./src/main/resources/test.txt"));
byte[] document2BA = FileUtils.readFileToByteArray(new File("./src/main/resources/test1.txt"));
byte[] document3BA = FileUtils.readFileToByteArray(new File("./src/main/resources/test2.txt"));

long //generatedocument1BAlength file for compound document content
File compoundFile = File.createTempFile("compound", ".bin");
OutputStream bos = new BufferedOutputStream(new FileOutputStream(compoundFile));
 
//partial document = Teildokument
String[] ranges = new String[3];                     //Byte ranges of the partial documents in the compound document
String[] partialNames = new String[3];        //Names of the partial documents
 
//write partial document bytestreams into binary compound file
long offset = 0;
long document1BAlength = document1BA.length;
String range1 = offset + "-" + (offset + document1BAlength - 1);
bos.write(document1BA);
ranges[0] = range1;
partialNames[0] = "test.txt";
 
offset += document1BAlength;
long document2BAlength = document2BA.length;
String range2 = offset + "-" + (offset + document2BAlength - 1);
bos.write(document2BA);
ranges[1] = range2;
partialNames[1] = "test1.txt";
 
offset += document2BAlength;
long document3BAlength = document3BA.length;
String range3 = (offset)= document1BA.length;
long document2BAlength = document2BA.length;
long document3BAlength = document3BA.length;

The converted individual files can easily be concatenated to an output stream as shown in the next example code block. Pay attention to set the offset correctly in order to not overwrite parts of individual contents. You should store the byte ranges that correspond to the individual original files. In the example, they are stored in the ranges array. Additionally, the original file names of the individual files are stored in the partialNames array. Those values can be used to provide a more convenient recognition of the referenced ranges in the sub-documents' metadata.

//generate file for compound document content
File compoundFile = File.createTempFile("compound", ".bin");
OutputStream bos = new BufferedOutputStream(new FileOutputStream(compoundFile));
 
//partial document = Teildokument
String[] ranges = new String[3];                     //Byte ranges of the partial documents in the compound document
String[] partialNames = new String[3];        //Names of the partial documents
 
//write partial document bytestreams into binary compound file
long offset = 0;
String range1 = offset + "-" + (offset + document3BAlengthdocument1BAlength - 1);
bos.write(document3BAdocument1BA);
ranges[20] = range3range1;
partialNames[20] = "test2test.txt";
 
IOUtils.closeQuietly(bos);

Creating the Metadata

Once the creation of the compound documents' content is complete, the corresponding metadata for the compound document and its subdocuments has to be generated for the import. The metadata of a compound document itself is no different from the metadata of any regular document - it consists of properties and a contentStreams object pointing to the binary file. The subdocuments, if they are to be imported together with the compound document, copy the contentStreams object of the compound document and add an attribute "range" to it, in which they enter the byte-digit interval recorded during content creation. 

{
    "objects": [
        {
            "contentStreams": [
                {
                    "fileName": "./compound.bin",
                    "mimeType": "application/octet-stream",
                    "cid": "cid_63apple"
                }
            ],
            "properties": {
                "objectTypeId": {
                    "value": "document"
     offset += document1BAlength;
String range2 = offset + "-" + (offset + document2BAlength - 1);
bos.write(document2BA);
ranges[1] = range2;
partialNames[1] = "test1.txt";
 
offset += document2BAlength;
String range3 = (offset) + "-" + (offset + document3BAlength - 1);
bos.write(document3BA);
ranges[2] = range3;
partialNames[2] = "test2.txt";
 
IOUtils.closeQuietly(bos);

Metadata for the Object Creation

Creating a Compound Document

The import endpoint POST /api/dms/objects expects a multipart request body. Multiple objects can be created in yuuvis® Momentum with one request. Thus, it is possible to create the compound document with the concatenated byte array as content file and, in the same request, some sub-documents.

As shown in the example objects list below, the compound document is defined first. All following objects are sub-documents. They refer to the same cid like the compound document, but a range is additionally specified. Here, the ranges correspond to the individual original files and their original file names are noted in the metadata of the sub-documents.

{
    "objects": [
        {
 },           "contentStreams": [
    "name": {           {
         "value": "testCompound"          "fileName": "compound.bin",
     }             }  "mimeType": "application/octet-stream",
     },         {      "cid": "cid_63apple"
     "contentStreams": [          }
      {       ],
             "fileNameproperties": "./compound.bin",{
                    "range": "0-1244","objectTypeId": {
                    "mimeTypevalue": "text/plaindocument",
                },
   "cid": "cid_63apple"            "name": {
   }             ],    "value": "testCompound"
        "properties": {       }
         "objectTypeId": {  }
        },
        {
"value": "document"              "contentStreams": [
  },                 "name": {
                    "valuefileName": "testcompound.txtbin",
                }    "range": "0-1244",
            }         }"mimeType": "text/plain",
        {            "cid": "contentStreamscid_63apple":
[                }
{            ],
        "fileName": "./compound.bin",   "properties": {
                "rangeobjectTypeId": "1245-2516",{
                    "mimeTypevalue": "text/plaindocument",
                },
    "cid": "cid_63apple"            "name": {
     }             ],  "value": "test.txt"
         "properties": {      }
          "objectTypeId": {  }
        },
        {
            "valuecontentStreams": "document"[
                {
   },                 "namefileName": {"compound.bin",
                    "valuerange": "test1.txt"1245-1338",
                }    "mimeType": "text/plain",
       }         }     ]
}

If subdocuments are to be imported later, the contentStreams object in the subdocument’s metadata comprises the contentStreamId and repositoryId from the DMS API response of the import of the compound document, the "mimeType" attribute befitting that subdocument and the same "range" attribute as with the concurrent import.

{
    "objects": ["cid": "cid_63apple"
                }
            ],
            "properties": {
             {   "objectTypeId": {
        "contentStreams": [           "value": "document"
        {        },
                "contentStreamIdname": "8FF6DBAE-1969-11E9-83A4-DFA1C5E44BD0",{
                    "repositoryIdvalue": "repo242test1.txt",
                }
   "range": "2517-3811",        }
        },
   "mimeType": "text/plain"    {
            }"contentStreams": [
           ],     {
       "properties": {                 "objectTypeIdfileName": {"compound.bin",
                    "valuerange": "document"
   1339-2610",
            },                 "name"mimeType": {"text/plain",
                    "valuecid": "test2.txtcid_63apple"
                }
            }],
        }     ]
}

Importing Compound Documents

...

"properties": {
                "objectTypeId": {
                    "value": "document"
                },
                "name": {
                    "value": "test2.txt"
                }
            }
        }
    ]
}

Creating Sub-Documents

It is also possible to create sub-documents of an already existing compound document. Do not assign again a binary content file, but reference the contentStreamId, repositoryId and archivePath where the binary content file of the compound document is stored. The archivePath is especially required if reconstruction is not possible with metadata information (e.g,, if a pathTemplate containing dynamic path elements like DATE is configured in the archive profile). As you can see in the example, you can also specify a concatenation of multiple ranges. 

{
    "objects": [
        {
            "contentStreams": [
                {
                    "contentStreamId": "8FF6DBAE-1969-11E9-83A4-DFA1C5E44BD0",
                    "repositoryId": "repo242",
                    "archivePath": "default/2023/01/06/",
                    "range": "1159-1365,0-45"
                    "mimeType": "text/plain"
                }
            ],
            "properties": {
                "objectTypeId": {
                    "value": "document"
                },
                "name": {
                    "value": "sub-concatenation.txt"
                }
            }
        }
    ]
}

Importing Compound Documents

Importing a compound document with the DMS API works in the same way as regular imports via POST of a multipart body with metadata and content to the endpoint /api/dms/objects. In the example, the compoundImportJsonString contains the metadata for the compound document and sub-documents as shown before.

RequestBody compoundImportRequestBody = new MultipartBody.Builder()
        .setType(MultipartBody.FORM)
        .addFormDataPart("data", "metaData.json", RequestBody.create(JSON, compoundImportJsonString))
        .addFormDataPart("cid_63apple", "compound.bin", RequestBody.create(OCTETSTREAM, compoundFile))
        .build();
 
Request compoundImportRequest = new Request.Builder()
        .header("Authorization", "Basic " + auth)
        .header("X-ID-TENANT-NAME", "default")
        .url(baseUrl + "objects")
        .post(compoundImportRequestBody)
        .build();
 
Response compoundImportResponse = client.newCall(compoundImportRequest).execute();

Importing Large Compound Documents

Importing very large compound documents together with the creation of many sub-documents (about 5000 or more) within one request can overload the Core API and cause some of the operations to fail. Therefore, it is recommended to stage such requests in several episodes: First, import the compound document itself as a single document and extract the contentStreamID and repositoryId from the response. From there, import the metadata of all sub-documents subsequently in a series of batch imports. Limit the amount of sub-documents per import to less than 5000 to avoid overloading. 

When subsequently importing sub-documents, please note that the metadata (postSubDocumentImportJsonString in the example) must be sent in a multipart body as well, even if this multipart body consists of only one part.

RequestBody compoundImportRequestBodypostSubDocumentImportRequestBody = new MultipartBody.Builder()
        .setType(MultipartBody.FORM)
        .addFormDataPart("data", "metaDatametadata.json", RequestBody.create(JSON, compoundImportJsonStringpostSubDocumentImportJsonString))
        .addFormDataPart("cid_63apple", "compound.bin", RequestBody.create(OCTETSTREAM, compoundFile))
        .build();


Request compoundImportRequestpostSubDocumentImportRequest = new Request.Builder()
        .header("Authorization", "Basic " + auth)
        .header("X-ID-TENANT-NAME", "default"tenant)
        .url(baseUrl + "objects")
        .post(compoundImportRequestBodypostSubDocumentImportRequestBody)
        .build();


Response compoundImportResponsepostSubDocumentImportResponse = client.newCall(compoundImportRequestpostSubDocumentImportRequest).execute();

Importing Large Compound Documents

Importing very large compound documents (5000+ subdocuments) can overload the Core API and cause some of the import operations to fail. Therefore, it is recommended to stage the import of such compound documents in several episodes: First, import the compound document itself as a single document and the contentStreamID and repositoryId are extracted from the response. From there, import the metadata of all subdocuments subsequently in a series of batch imports that limit the amount of subdocuments per import to less than 5000 to avoid overloading. 

When subsequently importing subdocuments, please note that the metadata must be sent in a multipart body when importing the subdocuments, even if this multipart body consists of only one part. 

...

...

Retrieving Sub-Documents

If you request the content of a compound document, you will get the total byte array. However, in most use cases, you want to retrieve a specific section of the total byte array. This is easily possible by retrieving a sub-document with a suitable range value that references the desired section in the total byte array (creation described above). You can retrieve its content by objectId (of the dub-document) as usual and you will get the corresponding section of the compound document's content.

Request getContentOfSubDocumentRequest = new Request.Builder()
        .header("Authorization", "Basic " + auth)
        .header("X-ID-TENANT-NAME", "default"tenant)
        .url(baseUrl .url(baseUrl + "objects/" + objectId + "objects/contents/file")
        .postget(postPartialDocumentImportRequestBody)
        .build();


Response postPartialDocumentImportResponsegetContentOfSubDocumentResponse = client.newCall(postPartialDocumentImportRequestgetContentOfSubDocumentRequest).execute();

>> Retrieving Documents via Core API

Deleting Compound Documents

Compound documents consist of one content document and several subdocuments that reference the content document through a content stream and one or more content ranges. When deleting these documents, it is generally avoided to remove any underlying content that may underlie other subdocuments. The subdocuments can be deleted as required, but the content remains accessible (unlike normal documents). If you request the deletion of a compound document, its metadata is deleted from the database. Thus, the DMS object itself does not exist in the system anymore. However, if at least one sub-document references a range within the former compound document's binary content file, the entire original byte array remains in the binary storage. It is even possible to define new sub-documents with reference on the same repositoryId and contentStreamId. As soon as you delete all sub-documents of an already deleted compound document, finally the binary content is deleted as well.

>> Deleting Documents via Core API

Summary

In this tutorial, we used an OkHttpClient with cookie handling to import to import and retrieve a compound document and several sub-documents through the Core API.

A complete code example can be found in this git repository.

...