Migration Config

Configuration File

GBDS Migration configuration parameters are defined in a configuration file, containing all parameters and its respective values. Omitted parameters assume their default values. This section describes the properties of the configuration file.

File Location

The configuration file is /etc/griaule/conf/gbds-migration/gbds-migration.properties.

File Properties

The configuration file must follow some requirements in order to be correctly interpreted by GBDS. These requirements are:

The file name and location must be exactly as mentioned in section 1.1;
There must be exactly one configuration parameter per line;
Each configuration parameter must be in the form <parameter>=<value>, without line breaks;
Each value must be separated by a comma when attributed to a single parameter.

Configuration Parameters

This section describes each GBDS Migration configuration parameter that can be listed in the configuration file, and how they affect the system operation.

General

gbscluster.zookeeper.quorum

Defines hostname and port through which zookeeper servers can be found. Each value must be separated by commas if more than one is available.

Default Value:

<hostname>:<port>

RDB Connection

gbds.rdb.driverClassName

Defines the class name for the relational database to be used for storing unsolved latents.

Default Value:

com.mysql.jdbc.Driver

gbds.rdb.url

Defines the relational database URL to be accessed.

Default Value:

jdbc:mysql://<hostname>:3306/gbds?useSSL=false&allowPublicKeyRetrieval=true

gbds.rdb.username

Defines the username to be used to access the relational database.

Default Value:

<rdb-username>

gbds.rdb.password

Defines the password to be used to access the relational database.

Default Value:

<rdb-base64-password>

gbds.rdb.dialect

Defines the dialect to be used in the relational database.

Default Value:

org.hibernate.dialect.MySQLDialect

gbds.rdb.showSql

Defines whether SQL statements must be included in the application logs.

Default Value:

false

Possible Values:

true
false

gbds.rdb.maxPoolSize

Maximum number of Connections a pool will maintain at any given time.

Default Value:

100

gbds.rdb.minPoolSize

Minimum number of Connections a pool will maintain at any given time.

Default Value:

1

gbds.rdb.initialPoolSize

Number of Connections a pool will try to acquire upon startup. This value must be in the range from gbds.rdb.minPoolSize to gbds.rdb.maxPoolSize.

Default Value:

2

gbds.rdb.maxStatments

Defines the size of c3p0's global PreparedStatement cache. If gbds.rdb.maxStatments is zero, statement caching will not be enabled.

This parameter controls the total number of Statements cached, for all Connections. If set, it should be a fairly large number, as each pooled Connection requires its own, distinct flock of cached statements. As a guide, consider how many distinct PreparedStatements are used frequently in your application, and multiply that number by gbds.rdb.maxPoolSize to arrive at an appropriate value.

Default Value:

0

gbds.rdb.maxIdleTime

Defines, in seconds, the time a Connection can remain pooled but unused before being discarded. Zero means idle connections never expire.

Default Value:

1800

gbds.rdb.maxConnectionAge

Defines, in seconds, the maximum time for a connection to live. A Connection older than gbds.rdb.maxConnectionAge will be destroyed and purged from the pool. This differs from gbds.rdb.maxIdleTime in that it refers to absolute age. Even a Connection which has not been much idle will be purged from the pool if it exceeds gbds.rdb.maxConnectionAge. Zero means no maximum absolute age is enforced.

Default Value:

1800

gbds.rdb.statementCacheNumDeferredCloseThreads

If set to a value greater than 0, the statement cache will track when Connections are in use, and only destroy Statements when their parent Connections are not otherwise in use. Although closing of a Statement while the parent Connection is in use is formally within spec, some databases and/or JDBC drivers, most notably Oracle, do not handle the case well and freeze, leading to deadlocks. Setting this parameter to a positive value should eliminate the issue. This parameter should only be set if you observe that attempts by c3p0 to close() cached statements freeze (usually you'll see APPARENT DEADLOCKS in your logs). If set, this parameter should almost always be set to 1.

Default Value:

1

gbds.rdb.acquireIncrement

Determines how many connections at a time c3p0 will try to acquire when the pool is exhausted.

Default Value:

10

gbds.rdb.testConnectionOnCheckout

If true, an operation will be performed at every connection checkout to verify that the connection is valid. Testing Connections in checkout is the simplest and most reliable form of Connection testing, but for better performance, consider verifying connections periodically using gbds.rdb.idleConnectionTestPeriod.

Default Value:

false

Possible Values:

true
false

gbds.rdb.testConnectionOnCheckin

If true, an operation will be performed asynchronously at every connection check-in to verify that the connection is valid. Use in combination with gbds.rdb.idleConnectionTestPeriod for quite reliable, always asynchronous Connection testing.

Default Value:

true

Possible Values:

true
false

gbds.rdb.acquireRetryAttempts

Defines how many times c3p0 will try to acquire a new Connection from the database before giving up. If this value is less than or equal to zero, c3p0 will keep trying to fetch a Connection indefinitely.

Default Value:

10

gbds.rdb.idleConnectionTestPeriod

If this is a number greater than 0, c3p0 will test all idle, pooled but unchecked-out connections, every this number of seconds.

Default Value:

30

HBase Column Families

Standard Column Family

These parameters are divided by biometric modality. They are column families used for template read operations.

gbds.hbase.templates.fingerprint.cf.name
gbds.hbase.templates.palmprint.cf.name
gbds.hbase.templates.face.cf.name
gbds.hbase.templates.iris.cf.name
gbds.hbase.templates.newborn-palmprint.cf.name

The default value for these parameters is tpts.

Fallback Column Family

These parameters refer to the column family previously used for storing the biometric templates, separated by biometric modality.

gbds.hbase.templates.fallback.fingerprint.cf.name
gbds.hbase.templates.fallback.palmprint.cf.name
gbds.hbase.templates.fallback.face.cf.name
gbds.hbase.templates.fallback.iris.cf.name
gbds.hbase.templates.fallback.newborn-palmprint.cf.name

The default values represent the column family used before changing these parameters and are, respectively: fingerprints, palmprints, faces, iris, newborn-palmprints.

GBDS Reextractor

General

gbds.reextract.nodeNumber

Node number running Reextractor. It determines scan range on HBase based on total nodes.

Default Value:

1

Minimum Value:

1

Maximum Value:

Value of gbds.reextract.totalNodes

gbds.reextract.totalNodes

Total nodes running Reextractor.

Default Value:

1

gbds.reextract.totalScanRegions

Total number of regions to break scans.

Default Value:

256 (00-FF regions)

gbds.reextract.scanners.number

Number of scanners. A scanner scans from HBase a range based on gbds.reextract.nodeNumber, gbds.reextract.totalNodes, and gbds.reextract.totalScanRegions.

Default Value:

5

gbds.reextract.workers.number

Number of workers. A worker holds a transaction template extraction.

Default Value:

5

gbds.reextract.writers.number

Number of writers. A writer gets the extraction result and writes back into transaction and people, if necessary.

Default Value:

5

gbds.reextract.range

Outer range configuration. It limits the automatic range.

Range can be a 2-char hex (such as 00 or A3) or a 2-char hex range (such as 00-01 or 4A-50).
Always 2-char hex.
If absent, GBS Reextractor will perform the automatic partition as usual.

gbds.reextract.validate.extraction

Flag to validate reextracted templates created previously.

On each transaction:
- When it is selected to extract, GBS Reextractor will not validate it.
- When it was extracted before, but not validated, GBS Reextractor will validated it.
- When it was extracted and validated, GBS Reextractor will skip it.
- Validation is saved on HBase on column transaction:<cf>-validated.
To ensure that a transaction is validated on a call after the extraction, remember to erase the SQLite file or to configure gbds.reextract.sqlite.resetOnStart=true. Otherwise the entire range that transaction belongs will be skipped.

This way reextraction and validation can be made on different GBS Reextractor calls, giving time to HBase write and consolidate templates on the transactions table.

Default Value:

true

Possible Values:

true
false

Pipeline Queue

gbds.reextract.workers.inqueueMaxSize

Queue size from scanner to worker. The greater this size, more scans are made and held on workers, but more memory is allocated.

Default Value:

100

gbds.reextract.writers.inqueueMaxSize

Queue size from worker to writer. The greater this size, more scans are made and held on writers, but more memory is allocated.

Default Value:

100

Modality to Reextract Flags

gbds.reextract.modality.finger

Determines if fingers should be reextracted.

Default Value:

true

Possible Values:

true
false

gbds.reextract.modality.face

Determines if faces should be reextracted.

Default Value:

true

Possible Values:

true
false

gbds.reextract.modality.palm

Determines if palms should be reextracted.

Default Value:

false

Possible Values:

true
false

gbds.reextract.modality.iris

Determines if irises should be reextracted.

Default Value:

false

Possible Values:

true
false

gbds.reextract.modality.newborn-palm

Determines if newborn palms should be reextracted.

Default Value:

false

Possible Values:

true
false

Template Extraction Microservices

gbds.reextract.msextraction.ginger.number

Defines how many fingerprint, palmprint, newborn, and sequence control template extraction microservice instances will be available. If this configuration is set to 0, the extraction microservices for these modalities will not be started.

Default Value:

10 (multiples of 10 recommended)

gbds.reextract.msextraction.face.number

Defines how many face template extraction microservice instances will be available. If this configuration is set to 0, the extraction microservices for this modality will not be started.

Default Value:

1 (10 times fewer than gbds.reextract.msextraction.ginger.number recommended)

gbds.reextract.msextraction.initialPort

This parameter defines the starting port number for the template extraction microservices.

Each microservice instance will increase its port number by 1. For example, considering the default port 6000, the first instance will use port 6000, the second will use port 6001, the third, 6002, and so on.

Do not conflict with API template extraction microservices ports (30000+), quality extraction microservices ports (31000) and GBDS matcher microservices ports (32000).

Make sure to allow the firewall ports that the microservice will use.

Default Value:

6000

gbds.reextract.msextraction.maxTries

Defines the maximum number of extraction tries GBDS will perform on a single biometric trait before returning an error.

Default Value:

3

gbds.reextract.msextraction.linkLibSegfault

Turns on/off segmentation fault lib debugger on the extraction microservice

Default Value:

true

Possible Values:

true
false

gbds.reextract.msextraction.checkTimeoutSecs

Timeout in seconds to check if the template extraction microservice is up.

Default Value:

30

gbds.reextract.msextraction.logLevel

Log level of the template extraction microservice.

Default Value:

INFO

Possible Values:

INFO
TIME
DEBUG

gbds.reextract.msextraction.timeout

Timeout in seconds for single call to the template extraction microservice.

Default Value:

60

gbds.reextract.msextraction.fingerprints.extractor.type

This parameter defines the type of ginger extractor preset to be used by GBDS Migration on mode --reextract, aka the GBDS Reextractor. Reextractor will save on HBase and RDB the ginger extractor type that was used the same way the API does, on HBase transactions column <cf>:ginger-extractor-type and on RDB transactions column ginger_extractor_type.

Default value:

GRIAULE_2024

Possible values:

GRIAULE_FAST: a simpler and faster version of GRIAULE_BASIC (never used on API).
GRIAULE_BASIC: default extraction for Verify.
GRIAULE_2020: old default extraction for Enroll, Update.
GRIAULE_2024: new default extraction for Enroll, Update.
GRIAULE_2018: use GRIAULE_2018 (slower).

Reextraction HBase Column Families

gbds.reextract.cf.finger

Finger column family name to receive extracted template on transactions and people.

Default Value:

fingerprint-reextract-1

gbds.reextract.cf.palm

Palm column family name to receive extracted template on transactions and people.

Default Value:

palmprint-write

gbds.reextract.cf.face

Face column family name to receive extracted template on transactions and people.

Default Value:

face-reextract-1

gbds.reextract.cf.iris

Iris column family name to receive extracted template on transactions and people.

Default Value:

iris-write

gbds.reextract.cf.newborn-palm

Newborn palm column family name to receive extracted template on transactions and people.

Default Value:

newborn-palmprint-write

SQLite

gbds.reextract.sqlite.filePath

File path for SQLite local database.

Default Value:

/home/<username>/reextract.db

gbds.reextract.sqlite.resetOnStart

Flag to reset SQLite on start.

Default Value:

false

Possible Values:

true
false

Configuration File Sample

Replace <hostname>, <rdb-username>, <rdb-base64-password>, and <username> with the correct values. Also, if zookeeper and mysql are running in ports different from the default ones, replace the port numbers.

# GENERAL
gbscluster.zookeeper.quorum=<hostname>:2181

# RDB CONNECTION
gbds.rdb.driverClassName=com.mysql.jdbc.Driver
gbds.rdb.url=jdbc:mysql://<hostname>:3306/gbds?useSSL=false&allowPublicKeyRetrieval=true
gbds.rdb.username=<rdb-username>
gbds.rdb.password=<rdb-base64-password>
gbds.rdb.dialect=org.hibernate.dialect.MySQLDialect
gbds.rdb.showSql=false
gbds.rdb.maxPoolSize=100
gbds.rdb.minPoolSize=1
gbds.rdb.initialPoolSize=2
gbds.rdb.maxStatments=0
gbds.rdb.maxIdleTime=1800
gbds.rdb.maxConnectionAge=1800
gbds.rdb.statementCacheNumDeferredCloseThreads=1
gbds.rdb.acquireIncrement=10
gbds.rdb.testConnectionOnCheckout=false
gbds.rdb.testConnectionOnCheckin=true
gbds.rdb.acquireRetryAttempts=10
gbds.rdb.idleConnectionTestPeriod=30

# HBASE COLUMN FAMILIES - STANDARD
gbds.hbase.templates.fingerprint.cf.name=tpts
gbds.hbase.templates.palmprint.cf.name=tpts
gbds.hbase.templates.face.cf.name=tpts
gbds.hbase.templates.iris.cf.name=tpts
gbds.hbase.templates.newborn-palmprint.cf.name=tpts

# HBASE COLUMN FAMILIES - FALLBACK
gbds.hbase.templates.fallback.fingerprint.cf.name=fingerprints
gbds.hbase.templates.fallback.palmprint.cf.name=palmprints
gbds.hbase.templates.fallback.face.cf.name=faces
gbds.hbase.templates.fallback.iris.cf.name=iris
gbds.hbase.templates.fallback.newborn-palmprint.cf.name=newborn-palmprints

# REEXTRACTOR - GENERAL
gbds.reextract.nodeNumber=1
gbds.reextract.totalNodes=1
gbds.reextract.totalScanRegions=256
gbds.reextract.scanners.number=5
gbds.reextract.workers.number=5
gbds.reextract.writers.number=5
gbds.reextract.validate.extraction=false

# REEXTRACTOR - PIPELINE QUEUE
gbds.reextract.workers.inqueueMaxSize=100
gbds.reextract.writers.inqueueMaxSize=100

# REEXTRACTOR - MODALITY TO REEXTRACT FLAGS
gbds.reextract.modality.finger=true
gbds.reextract.modality.face=true
gbds.reextract.modality.palm=false
gbds.reextract.modality.iris=false
gbds.reextract.modality.newborn-palm=false

# REEXTRACTOR - TEMPLATE EXTRACTION MICROSERVICE
gbds.reextract.msextraction.ginger.number=10
gbds.reextract.msextraction.face.number=1
gbds.reextract.msextraction.initialPort=6000
gbds.reextract.msextraction.maxTries=3
gbds.reextract.msextraction.linkLibSegfault=true
gbds.reextract.msextraction.checkTimeoutSecs=30
gbds.reextract.msextraction.logLevel=INFO
gbds.reextract.msextraction.timeout=60
gbds.reextract.msextraction.fingerprints.extractor.type=GRIAULE_2024

# REEXTRACTOR - REEXTRACTION HBASE COLUMN FAMILIES
gbds.reextract.cf.finger=fingerprint-reextract-1
gbds.reextract.cf.palm=palmprint-write
gbds.reextract.cf.face=face-reextract-1
gbds.reextract.cf.iris=iris-write
gbds.reextract.cf.newborn-palm=newborn-palmprint-write

# REEXTRACTOR - SQLITE
gbds.reextract.sqlite.filePath=/home/<username>/reextract.db
gbds.reextract.sqlite.resetOnStart=false

Last updated 29 days ago