DCS SPOBT Offline Users Upload and Download

Technical Solution

Author: Shmulik Kachlon

Head of development and Technology

Revision History
21/09/200 Document Created Shmulik Kachlon
9
25/09/200 Staging database diagram Shmulik Kachlon
9 was added
28/09/200 Defining the solution and Shmulik Kachlon
9 describing the prison
responsibilities
29/09/200 Solution design diagram was Shmulik Kachlon
9 added
20/10/200 Stipulating the action should Shmulik Kachlon
9 be taken when offline upload
is being carried out
26/11/200 Changing the staging Shmulik Kachlon
9 database schema and
elaborating on the download
process

www.argus-solutions.com 21/09/2009 Page 1 of 10

DCS SPOBT Offline Users’ Upload and Download
Table of Contents
1 General.................................................................3
1.1 Acronym..................................................................................................................................3
1.2 Audience..................................................................................................................................3
1.3 Purpose...................................................................................................................................3
1.4 Initial Upload Design Diagram.....................................................................................................3
2 Technical Solution......................................................5
2.1 Reason for defining the solution..................................................................................................5
2.2 Solution’s concept.....................................................................................................................5
2.3 Proposed Solution.....................................................................................................................5
2.4 CC Responsibility and Interaction................................................................................................9
3 Configuring the Spobt before running the Offline Tool..................10

www.argus-solutions.com 21/09/2009 Page 2 of 10

DCS SPOBT Offline Users’ Upload and Download
1 General

1.1 Acronym

SPOBT Single Point Of Biometric Truth

CC Correctional Centre
AS Argus Solutions
DCS Department of Corrective Services
BI Biometric Innovations

1.2 Audience

The audience of this document is every third party system which needs to upload
and download users in an offline mode to and from Spobt to cater for bulk upload
and download.

1.3 Purpose

The purpose of this document is to outline the technical solution for an offline bulk
upload of users from a CC to the Spobt and the creation of the downloadable
users’ storage for the new prison in order to update its local data storage.

The reason for having a separate method for uploading CC users through a
different interface but the Spobt service is the volume of the data that will be
uploaded into the Spobt. The Spobt service is not designed to support such a
massive upload operation hence AS had to design another solution for uploading
bulk of users’ information through a software which will run in the Spobt server
directly for saving the huge traffic on the DCS internal network.

This solution will also serve as an offline tool for synching users with the Spobt
server in case of disconnection of prisons from the Spobt server for awhile for
whatever reason.

1.4 Initial Upload Design Diagram

The following diagram shows the software solution design by which the CC users’
information is captured at the CC level and then copied across to the Spobt:

www.argus-solutions.com 21/09/2009 Page 3 of 10

DCS SPOBT Offline Users’ Upload and Download
 www.argus-solutions.com 21/09/2009 Page 4 of 10
DCS SPOBT Offline Users’ Upload and Download
1 Technical Solution

The section describes the technical implementation for uploading users for the very
first time to the Spobt by a CC.
The outcome of the process is described as well so that a CC could take it and
applies its local rules on it and persists the Spobt’s users it might not have locally on
its own database.

2.1 Reason for defining the solution

The main reason for designing another method for uploading the initial bunch of users
from a CC to the Spobt is the potential large amount of data which should be replicated
from the local CC database to the Spobt database.
If the Spobt’s service would have used for this purpose, there was no guarantee it would
be able to cope with this amount of users and the DCS network would have throttled if we
did so.

Therefore, Argus has to define another way of uploading bulks of users from a CC to the
Spobt server through a staging process to make it works faster and more efficient and
more important, feasible.
2.2 Solution’s concept

AS explored few possible technical solutions and tested few of them and made a decision
to put forward the proposed solution for it has the better test outcomes.

The concept of the solution is to enable prisons to collect users from their own local
databases, store them in a well known format by the Spobt server offline sync software
utility so that the Spobt will store it on the central database and replicate them to any CC
which is subscribed to the Spobt services.

2.3 Proposed Solution

The technical solution consists of the following software components:

• A software utility which is provided by the CC workflow system provider (at the
time of writing this documents the providers are AS and BI)
• A staging database which consists of six data tables to store all the information for
the user required by the Spobt
• AS Spobt initial upload software utility to extract the users from the staging
database provided by the CC. Once upload has finished, a download operation will
start to create a new staging database for the prison to be stored on the same hard
drive provided by the CC, extract the users from that database and persist the
users to the local workflow system’s database (AS has developed a tool for its
internal use. Any consumer of the staging database should cater for its own
persistence to the local users’ storage)

2.3.1 Collecting the CC’s local users

Collecting the user is the process of getting all the users which are currently stored
in the local CC database. It is the responsibility of the CC to collect its users and
have them stored in the staging database in the right format.

www.argus-solutions.com 21/09/2009 Page 5 of 10

DCS SPOBT Offline Users’ Upload and Download
2.3.2 Persisting the CC local Users to an Upload Staging Database

Once collected, the users will have to be stored in the staging database (defined in
point 2.3.5) and populate the six tables with all the data which is relevant to the
Spobt central database.
The storage in the staging database is the responsibility of the workflow system
provider .
The AS offline upload and download software utility (point 2.3.4) will then connect
to this staging database , extracts the users from it, persists them to the Spobt
central server and update the UploadResult table (see diagram in 2.3.5) with the
external system user id (i.e. user ids provided by either Cornerstone or
DoorKeeper) alongside the corresponding data centre user id for each external
user.
The prison should then update its local database with the data centre user id for
each local user for further linking between the Spobt’s users and each prison’s
users.

If an exception has occurred during an upload of a user, the UploadResult table will
have the error information for this specific user attached to external user system id
(i.e. Cornerstone user id or DoorKeeper user id).

2.3.3 Persisting the Spobt’s existing Users to a Download Staging Database

Once the upload operation has finished, the users are now in the Spobt central
database available for other prisons.
The AS offline sync utility tool for the Spobt will then start the download operation
for each prison which is subscribed to the Spobt.

This has to be done for two reasons:

1. The new prison should get the existing Spobt users that were uploaded by other
prisons up until this point
2. All the existing prisons subscribed to the Spobt should get the new users that
have just been uploaded by the Spobt ‘s newly introduced prison.

The AS Spotb’s offline sync tool will create a staging database for each prison with
all the relevant users to this prison. E.g., if there are existing 4 prisons subscribed
to the Spobt and a new prison has just subscribed to it, the output of running the
AS Spotb’s offline sync tool will be as follow:
• An updated upload staging database with the data centre user ids
correspond to the uploaded users as well as errors in the event they occur
(this data is stored in the UploadResult table)
• Five new staging databases with the download users for each prison in the
Spobt (the four existing ones and the just added one)

2.3.4 AS Spobt’s Offline Sync Software Tool

The AS offline upload and download tool is a .net win form application which is
divided into two sections:
11 The upload section for the new prison’s users
11 The download section for creating a staging download database for the new
added prison and the existing prisons in the Spobt

www.argus-solutions.com 21/09/2009 Page 6 of 10

DCS SPOBT Offline Users’ Upload and Download
 The upload section takes care of uploading the users from the staging database
which is provided by the new prison that was just introduced to the Spobt.
Once the upload has been completed, there is a process of setting the data centre
user id against each external system user id (i.e. the user id of the user in either
Cornerstone or DoorKeeper) into the UploadResult data table.

After this has finished, a download operation can start. The download will populate
a new clean staging database for the new prison and for each existing prison in the
Spobt.
There is one difference between the download staging databases which are created
for the new prison and for the existing prisons in the Spobt:
11 For the new prison, the UploadResult table will be populated with the
ExternalSystemUserKey alongside the DataCentreUserId. For the existing
prisons the UploadResult table will be empty.

2.3.5 Exception handling for uploading users

Exception handling will be done exactly the same way it is done through a normal
upload user operation using the Spobt service.
In case of exception\error while uploading a user through the AS offline upload and
download tool, the exception will be captured in the UploadResult table using the
Error column and the IsSuccessful column will be set to false.
That means, when the new prison gets back a staging database created by the AS
offline upload and download tool, when extracting the user information for updating
the data centre user id locally in the prison’s database, the prison should check the
IsSuccessful columg on the UploadResult table. If True, then a data centre user id
will be provided for the uploaded user. If False, the Error column will specify the
error which has occurred and the data centre user id will be empty.
The prison will then have to send the failed to upload users again through the
normal upload notification received by the Spobt service.

2.3.6 Staging Database Diagram

The staging database schema consists of six tables. This schema is the same for both
upload and download.

The following figure is the staging database schema:

www.argus-solutions.com 21/09/2009 Page 7 of 10

DCS SPOBT Offline Users’ Upload and Download
 www.argus-solutions.com 21/09/2009 Page 8 of 10
DCS SPOBT Offline Users’ Upload and Download
Note: The User data table encapsulates all the demographic data of the user. For every
optional field, there is a Is+field name+Deleted column attached to it to mark whether this
field was deleted or not. This applies exactly the same rules provided in the API document
regarding the Deletable objects.

2.4 CC Responsibility and Interaction

The prison is responsible for taking care of the following:

1. Providing the upload staging database to Argus so that Argus can run its offline
synchronization software tool in order to upload the new users locally on the Spobt
server
2. Extracting the data centre user ids from the UploadResult table and set it locally in
the prison’s database next to the local user id. The data centre id for each user will
be set through the offline upload process.
3. Extract the users’ information from the download staging database and store it
locally in the prison database. The download staging database has exactly the
same database schema of the upload staging database. The purpose of this
database is to provide the new prison all the users that were uploaded by the other
prisons in the Spobt before the new prison has subscribed to the Spobt.
This database is likely to be very big and will have to be extracted locally at the
prison (potential number of users can reach hundreds of thousands).

The above mentioned are the equivalent operations for synching users with the Spobt
service but in an offline mode running locally on the Spobt server without having to send
big amount of data across the DCS network.

www.argus-solutions.com 21/09/2009 Page 9 of 10

DCS SPOBT Offline Users’ Upload and Download
1 Configuring the Spobt before running the Offline Tool

As the offline upload is a very time consuming operation, the following stipulations have to
be taken into account:

1. A notice should be sent to all the prisons that are active against the Spobt few
days in advanced regarding the impending offline upload of the newly introduced
prison
2. The Spobt Broadcaster service should be stopped for not propagating any Spobt’s
availability messages to any prison
3. All the prisons which are subscribed to the Spobt should become inactive through
the Spobt Web Admin Tool
4. Upon completion of the offline upload, all the prisons should be reactivated through
the Spobt Web Admin Tool
5. All the prisons should be notified they are back in business
6. The Spobt Broadcaster Service has to be restarted for sending auto notification to
the subscribed prisons for normal operation to be continued

www.argus-solutions.com 21/09/2009 Page 10 of 10

DCS SPOBT Offline Users’ Upload and Download