...

...

Unforeseen circumstances might occur in any system and might lead to differences between database and Elasticsearch index. As an example, a deleted object was correctly removed from the database, but for whatever reason not from the Elasticsearch index. As of yuuvis® Momentum version 2020 Winter, it is possible to check the consistency of the stored objects by means of the maintain compare command provided by the Commander service.

The command determines the objects that were last modified (or created) within a specified period of time and for both storage locations, database and Elasticsearch index. For every single day and for each single object, its properties are compared between database and Elasticsearch index. Differences are summarized and returned in a JSON list as shown in the example on the bottom of the articleWhile the command runs continuously, the comparison of DMS objects is performed step by step. The objects are grouped by their last modification date/creation date. For each day between the start date and the end date, the corresponding objects are processed. Then again, each day is split in a variable number of time periods, such that the amount of objects to be compared in each period is approximately equal to the number of fetch-size.

All results will be summarized in a file called job-result.json. In this (ND)JSON structed file, each day is listed separately and contains some general comparison results.

{"1970-01-01" : {"total":{"missingDb":10,"missingEs":5,"differences":2,"dbHits":20,"esHits":20},"end":"1970-01-02T00:00:00.000Z","start":"1970-01-01T00:00:00.000Z","daily":{"comparison":{"propertyDifferences":2,"missingInDb":10,"missingInEs":5},"splits":1,"esHits":20,"dbHits":20}}}
{"1970-01-02" : {"total":{"missingDb":15,"missingEs":10,"differences":4,"dbHits":40,"esHits":40},"end":"1970-01-03T00:00:00.000Z","start":"1970-01-02T00:00:00.000Z","daily":{"comparison":{"propertyDifferences":2,"missingInDb":5,"missingInEs":5},"splits":1,"esHits":20,"dbHits":20}}}

For every day with differences, there is an additional folder, containing some more detailed information about this day's differences.

{
	"timeRangeSummary": {
		"start": "1970-01-01T00:00:00.000Z",
		"end": "1970-01-02T00:00:00.000Z",
		"comparison": {
			"missingInDb": [
			"00000000-0000-0000-0000-0000000000000",
			"00000000-0000-0000-0000-0000000000001",
			"00000000-0000-0000-0000-0000000000002",
			"00000000-0000-0000-0000-0000000000003",
			"00000000-0000-0000-0000-0000000000004",
			"00000000-0000-0000-0000-0000000000005",
			"00000000-0000-0000-0000-0000000000006",
			"00000000-0000-0000-0000-0000000000007",
			"00000000-0000-0000-0000-0000000000008",
			"00000000-0000-0000-0000-0000000000009",
			"00000000-0000-0000-0000-0000000000010"

			]
			"missingInEs": [
            "00000000-0000-0000-0000-0000000000011",
			"00000000-0000-0000-0000-0000000000012",
			"00000000-0000-0000-0000-0000000000013",
			"00000000-0000-0000-0000-0000000000014",
			"00000000-0000-0000-0000-0000000000015",
			"00000000-0000-0000-0000-0000000000016"
			],
			"differences": [
			"00000000-0000-0000-0000-0000000000017",
			"00000000-0000-0000-0000-0000000000018"
			]
		}
	}
}

After all time periods of a day are finished, the procedure is repeated for the next day until the current date (or the specified end date) has been processed. Depending on the number of objects for the tenant, this process can take a long time. While running the task, the process progress is continuously displayed on the command line including an estimated completion time.

When the command finished, the directory containing all result files will be shown as result.

The following parameters can be given to the maintain compare command:

The following modes can be selected for the maintaincompare command:

The following code block shows an example response. The job section contains information on the parameters that were specified for the command. The next sections are named after the dates for which differences in the analyzed object properties were found. The MissingInES list contains the object IDs specifying the objects that were found in the database but not in the Elasticsearch index. Objects specified by IDs listed in Differences were found in both storage locations, but at least one of the analyzed properties has a different value. The MissingInDb list contains the object IDs specifying the objects that were found in the Elasticsearch index but not in the database.

...

...

...