The goal of this project was to design an efficient data anonymization system for preparing test environments.
When test environments are created by copying part or all of the production database for development and QA purposes, sensitive personal data subject to privacy regulations must be anonymized — meaning altered in such a way that identifying the data subject is impossible without additional information.
To meet this requirement, various data masking algorithms can be applied. The key condition: the original data must not be recoverable.
Taking into account both these and the client’s additional requirements for performance, we developed a custom anonymization procedure for an Oracle 19 database.
Average anonymization speed: 1.5 TB/hour.
The high performance was achieved through the use of:
- Efficient in-place masking architecture (via UPDATE operations), requiring only one environment. Unlike traditional source/target (SELECT/INSERT) schemes that require two environments during execution.
- Flexible parallelism using built-in Oracle tools, optimized based on the storage subsystem and database server capacity.
- Selective disabling of indexes/triggers during execution to minimize rebuild time afterward.
- Optimized data masking methods tailored to different data types.
Additional advantages:
- Automated comparison of DDL/status of database objects (indexes, constraints, triggers) before/after masking and against a reference DB.
- Full logging of operations, enabling troubleshooting and providing detailed execution progress.
- Where required, anonymized values preserve valid formats (e.g., credit card numbers, tax IDs, emails) for downstream system compatibility.
