TAG LINE
TAG LINE
SMALL TITLE

Archiving Data

Last Updated: Tue Dec 13 2022

Overview

The LogiSense Billing application has a built in data archival tool which removes old data from the system to prevent its database from growing too large and prevent certain tables from becoming bloated.

Data that is archived is sent to an S3 bucket and organized by day and target (targets are explained below).

Archived Data and Targets

The table below details data archive targets and recommended settings for them. These settings can be modified per target if required. The general default is to store data in a production system for 3 months.

Note: There are certain targets that cannot have their archive type changed (for example, API Logs currently can’t be archived, they must be purged).

Release

Target

Description

Archive Type and Duration

Reason

10.4.0

Usage

Data relating to Usage Records (Rated, Mediated, Extensions, Aggregates).

Archive & Purge - 3 Months

Usage Records that were processed could be audited at a later date, by auditors or customer disputes.

10.4.0

Unprocessed Usage Exceptions

The usage exceptions that have not been resolved.

Purge - 3 Months

If the exceptions haven’t been dealt with after 3 months, they probably aren’t going to be resolved or charged.

10.4.0

Processed Usage Exceptions

Usage Exception records that have been sent for Rerating.

Purge - 1 Day

These would normally get deleted once sent for reprocessing.

10.4.0

Rerated Usage Records

Original Usage Records that have been sent in for Rerating, making the original record obsolete.

Purge - 1 Day

These would normally get deleted once sent for reprocessing.

10.4.0

API Logs

The request/response API logs from every API call made to the system.

Purge - 5 days

There are many API calls made every day in the system and this table fills up quickly. This data also is rarely referenced again.

10.4.0

Tax Logs

The Tax Logs from every tax calculation made in the system.

Archive & Purge - 3 Months

Tax calculations are audited annually and could be challenged on any invoice.

10.4.0

Webhook Logs

The response data from every webhook sent from the system.

Purge - 3 Months

These generally are not required once the change has been made or the action has completed.

10.5.0

Data Import Tracking and Batch details

Usage API and Data Import Tracking details

Purge - 3 Months

Once imported, this data is used by the system from the UDRMediated and UsageRated tables. The import details are no longer used by the system.

Monitoring Data Archival Activity

Within the LogiSense Billing AdminPortal data archiving activity can be tracked by loading the Operations / Monitor / Data Archive screen.

All data archive operations are by default configured to run for a maximum of six hours. If an operation takes more than six hours it will be cancelled and listed as Cancelled on this screen.

Downloading Archived Data from S3

When data is successfully archived to an S3 bucket it will be listed as Completed on the Data Archive screen and the archived files will be listed under the Files tab. From this tab you can download the archived files if you are required to keep them longer than they will be stored in the S3 bucket.