Appending Rows

During an incremental aggregate build, AtScale retrieves rows from the fact dataset that have an incremental indicator value greater than the last value recorded in the incremental aggregate.

Example

Suppose that the last incremental indicator key value used in the incremental aggregate is 1475191168000 (the UNIX timestamp in seconds since epoch for Thu, 29 Sep 2016 23:19:28 GMT). When an incremental rebuild of the aggregate begins, the engine will look in the fact dataset for rows with an incremental indicator value greater than this value, perform the necessary data aggregation, and append rows to the aggregate.

Note: When the incremental indicator column type is as a timestamp, the grace period is always interpreted as Seconds.

Updating Rows

You also specify a grace period when you select a fact dataset's column to use as an incremental indicator. When the AtScale engine starts an incremental build, the grace period determines how far back in time the engine will look for updates to rows in the fact dataset, based on the units that are in the indicator. The engine then updates rows in the aggregate table where the incremental indicator matches those of rows in the fact dataset.

At the start of an incremental rebuild, the AtScale engine runs a query against an aggregate to find the latest incremental indicator to figure out new data that has been added since the last update. The query uses a WHERE filter to only scan data since the last update which enables table pruning against partitioned fact datasets.

Moreover, if more than one aggregate configured for incremental rebuilds is based on the same fact datasets, the engine runs the query to see what new data has been added since the last update and uses the result for the remaining aggregates. For example, suppose that six aggregates configured for incremental rebuilds are based on the table factinternetsales. The query is run on the first of these six aggregates and the result is used for all six. This process saves time and resources versus the alternative of running the query for each aggregate.

Example

As in the previous example, suppose that 1475191168000 is the last incremental indicator key value in an aggregate table. The grace period is 86400 seconds (one 24-hour period). The engine looks back in the fact dataset to the first row where the indicator is 86400 seconds earlier than the moment that the incremental build began. The engine also does this in the aggregate table.

Next, the engine compares the data in the fact set with the aggregated data in the aggregate table, moving forward in time in the data until the engine reaches the point in time when the incremental build began. As it compares data, the engine updates the aggregated data when it finds changes in the fact dataset.