Compare the objects stored in the database and Elasticsearch index.

Unforeseen circumstances might occur in any system and might lead to differences between database and Elasticsearch index. For example, a deleted object was correctly removed from the database, but for whatever reason not from the Elasticsearch index. As of yuuvis® Momentum version 2020 Winter, it is possible to check the consistency of the stored objects by means of the maintain compare command provided by the Commander service.

While the command runs continuously, the comparison of DMS objects is performed step by step. The objects are grouped by the date of their last modification/creation. For each day between the start date and the end date, the corresponding objects are processed. Then again, each day is split in a variable number of time periods, such that the amount of objects to be compared in each period is approximately equal to the number of fetch-size.

All results are summarized in a file called job-result.json. In this (ND)JSON structed file, each day is listed separately and contains some general comparison results.

Example 'job-result.json' file

{"1970-01-01" : {"total":{"missingDb":10,"missingEs":5,"differences":2,"dbHits":20,"esHits":20},"end":"1970-01-02T00:00:00.000Z","start":"1970-01-01T00:00:00.000Z","daily":{"comparison":{"propertyDifferences":2,"missingInDb":10,"missingInEs":5},"splits":1,"esHits":20,"dbHits":20}}}
{"1970-01-02" : {"total":{"missingDb":15,"missingEs":10,"differences":4,"dbHits":40,"esHits":40},"end":"1970-01-03T00:00:00.000Z","start":"1970-01-02T00:00:00.000Z","daily":{"comparison":{"propertyDifferences":2,"missingInDb":5,"missingInEs":5},"splits":1,"esHits":20,"dbHits":20}}}

For every day with differences, there is an additional folder, containing some more detailed information about this day's differences.

Example folder content

{
	"timeRangeSummary": {
		"start": "1970-01-01T00:00:00.000Z",
		"end": "1970-01-02T00:00:00.000Z",
		"comparison": {
			"missingInDb": [
			"00000000-0000-0000-0000-0000000000000",
			"00000000-0000-0000-0000-0000000000001",
			"00000000-0000-0000-0000-0000000000002",
			"00000000-0000-0000-0000-0000000000003",
			"00000000-0000-0000-0000-0000000000004",
			"00000000-0000-0000-0000-0000000000005",
			"00000000-0000-0000-0000-0000000000006",
			"00000000-0000-0000-0000-0000000000007",
			"00000000-0000-0000-0000-0000000000008",
			"00000000-0000-0000-0000-0000000000009",
			"00000000-0000-0000-0000-0000000000010"

			]
			"missingInEs": [
            "00000000-0000-0000-0000-0000000000011",
			"00000000-0000-0000-0000-0000000000012",
			"00000000-0000-0000-0000-0000000000013",
			"00000000-0000-0000-0000-0000000000014",
			"00000000-0000-0000-0000-0000000000015",
			"00000000-0000-0000-0000-0000000000016"
			],
			"differences": [
			"00000000-0000-0000-0000-0000000000017",
			"00000000-0000-0000-0000-0000000000018"
			]
		}
	}
}

After all time periods of a day are processed, the procedure is repeated for the next day until the current date (or the specified end date) is reached. Depending on the number of objects for the tenant, this process can take a long time. While running the task, the process progress is continuously displayed on the command line including an estimated completion time.

After the command finishes, the directory containing all result files is shown.

The following parameters can be given to the maintain compare command:

Parameter	Description	Required	Default	Example
`-t \| --tenant <arg>`	Specifies the target tenant. If not specified, the command is executed for all tenants.	yes	-	`maintain-compare -t default`
`--target-directory` `<arg>`	Target directory for result log.	no	.	`maintain-compare -t default --target-directory C:\Users\myusername\ymtenantdeletionlog`
`-s \| --start-date <arg>`	Specifies the start date for the comparison in `yyyy-MM-dd` format. Document objects with a `system:lastModificationDate` (or `system:creationDate`, see parameter `-c`) later than the specified date at `00:00:00.000` are taken into account for the comparison. If not specified, the earliest `system:lastModificationDate` (or `system:creationDate`) for an object found in the database or Elasticsearch index is used as start date.	no	firstobjectcreation	`maintain-compare -t default -s 2020-11-11`
`-e \| --end-date <arg>`	Specifies the end date for the comparison in `yyyy-MM-dd` format. Document objects with a `system:lastModificationDate` (or `system:creationDate`) before and inclusive of the specified date at `23:59:59.999` are taken into account for the comparison. If not specified, the current date is used as end date.	no	today	`maintain-compare -t default -s 2020-11-11 -e 2020-11-12`
`--fetch-size <arg>`	Fetch size, used to split days in time periods. Each period contains approximately as many elements as specified by `fetch-size`. The value must be at least 1000 and not bigger than 50,000.	no	10000	`maintain-compare -t default` --fetch-size 20000
`-c \| --useCreationDate`	If the parameter is set, the `system:creationDate` is used instead of the `system:lastModificationDate` to decide whether the corresponding object is included in the comparison.	no	false	`maintain-compare -t default -s 2020-11-11 -e 2020-11-12 -c`
`-m \| --mode <arg>`	Specifies the mode of the comparison determining which object properties are included in the comparison. The available modes are listed and described in the table below.	no	SIMPLE	`maintain-compare -t default -s 2020-11-11 -e 2020-11-12 -m ADVANCED`

The following modes can be selected for the maintain compare command:

Mode	Description
`COUNT`	The command determines and compares the number of objects modified (or created) on every single day in the specified time range.
`SIMPLE`	The command determines and compares only the `system:objectId` of all objects that are included in the comparison.
`BASIC`	The command determines and compares the following properties of all objects that are included in the comparison: `system:objectId` `system:createdBy` `system:createdBy` `system:creationDate` `system:lastModifiedBy` `system:lastModificationDate` `system:tenant` `system:versionNumber` `system:deleted`
`ADVANCED`	The command determines and compares the properties listed for the `BASIC` mode and additionally the following properties of all objects that are included in the comparison: `system:baseTypeId` `system:objectTypeId` `system:traceId` `system:tags`

Read on

COMMANDER Service for System Maintenance

Perform low-level maintenance on your core system, access the database and carry out Elasticsearch queries. Keep reading

dbs-reindex Command

Restore the entire Elasticsearch index from the database via Commander service. Keep reading

maintain-repository Command

Find content files in the S3 store that are not referenced any longer by an object in the database. Keep reading

Browser not supported