Load handling

Maximum Number of Simultaneous Workflow Clients Through REST

Overview

Configuring many systems to perform a full system scan using MClient.

Queuing Mechanism

We will have 2 queues in REST.

  1. Pending: Pending scans

    1. Scan not yet submitted to the COM service.

    2. Maximum 20,000 files here before server returns 500 too busy.

    3. Files go pass through here first, then if the ongoing scan queue has space they are moved there.

  2. Ongoing: Scan in progress

    1. Scan already submitted to the COM service.

    2. Maximum 1000 files here.

    3. If this queue is full the files stay in pending scan queue.

Disk Check

We have 2 types of disk checks based on how much free space is left.

  1. If < 3.5 GB free disk space:

    1. With every file upload we will do a disk space check.

  2. If > 3.5 GB free disk space:

    1. We used the cached value from our already periodic check.

The reason we need to checks is because there is a known issue about Mongo not being able to start if there is enough free space on the disk. (3379MB)

Mongo Error
2016-09-06T16:12:15.514-0700 [initandlisten] ERROR: Insufficient free space for journal files
2016-09-06T16:12:15.514-0700 [initandlisten] Please make at least 3379MB available in C:\Program Files (x86)\OPSWAT\Metadefender Core 4\Mongo\data\journal or use --smallfiles

Recommend REST integration in multiple client environment

  1. REST users should only submit ~20 files at a time and wait until processing is complete before uploading more.

  2. /file/inqueue reports how many files are in the queues (adding both pending on ongoing)

  3. If client gets 503 server too busy they should wait a bit and try again.