Usage method
Call of pghist.hist_enable procedure creates 3 objects in the table schema for working with history:
1. [schema].[table]_hist view is a change log (audit) for developers to analyze data problems
SQL query and application name allow to determine the location in the source code, transaction identifier - changes in other tables and SQL queries.
Example:
-- Create schema and table, enable history, change data drop schema if exists example cascade; create schema example; create table example.document( id int primary key, number varchar(10), amount numeric(10,2) ); call pghist.hist_enable('example', 'document'); insert into example.document values (11, '#10', 100); insert into example.document values (12, '#20', 200); update example.document set number='#20/2',amount=210 where id=12; update example.document set amount=220 where id=12; delete from example.document where id=11; -- Select full info select * from example.document_hist; -- Select partial info by id select hist_timestamp,hist_operation,hist_db_user,amount_old from example.document_hist where id=12 and (hist_operation!='UPDATE' or 'amount'=any(hist_update_columns)) order by hist_statement_id; -- Select partial info with current values select * from example.document_hist union all select null,null,null,'[CURRENT_VALUES]',null,null,null,null,null,null,* from example.document order by id,1; -- Create extended view and grant select on it to developer -- (alternatively to 'grant select on all tables in schema pghist to developer'); select * from pghist.hist_data$example_document h join pghist.hist_statement s on s.id = h.hist_statement_id join pghist.hist_query q on q.hash = s.query_hash join pghist.hist_transaction t on t.id = s.transaction_id order by h.hist_statement_id, h.id; create or replace view example.document_hist_ext as select h.id,t.timestamp_commit,t.db_client_addr,q.text query_text from pghist.hist_data$example_document h join pghist.hist_statement s on s.id = h.hist_statement_id join pghist.hist_query q on q.hash = s.query_hash join pghist.hist_transaction t on t.id = s.transaction_id order by h.hist_statement_id, h.id; select * from example.document_hist_ext where id=12; grant select on example.document_hist_ext to developer_1;
2. [schema].[table]_changes function returns a set of data to display changes to the user
Information is displayed in the old-new value format with a description.
Display identifiers and foreign keys to the user is uninformative, so for each row and field values description is displayed that can be overridden via SQL expression. By default, comment table with the primary key is returned for a row description, first text field of related table is returned for a column that is the foreign key, for others - value. Thus, can display to user a description without value.
When enabling history for detail table, must specify the master table, foreign key values will be stored and indexed. By default, when changes are received in the master table, changes in the detail tables will also be returned.
Some columns (for example, username) can be redefined via customization functions.
Example:
-- Create schema and tables drop schema if exists example cascade; create schema example; create table example.customer( id int primary key, name varchar(100) not null ); insert into example.customer values (1,'Horns'),(2,'Hooves'); create table example.invoice( id int primary key, number varchar(10), date date, customer_id int references example.customer(id), amount numeric(20,2) ); comment on table example.invoice is 'Invoice'; comment on column example.invoice.id is 'Identifier'; comment on column example.invoice.number is 'Number'; comment on column example.invoice.date is 'Date'; comment on column example.invoice.customer_id is 'Сustomer'; comment on column example.invoice.amount is 'Amount'; create table example.product( id int primary key, name varchar(100) not null, code varchar(10) not null ); create table example.invoice_product( id serial primary key, invoice_id int references example.invoice(id), product_id int references example.product(id), quantity int, color char(1) check (color in ('R','G','B')) ); comment on table example.invoice_product is 'Product of invoice'; comment on column example.invoice_product.id is 'Identifier'; comment on column example.invoice_product.invoice_id is 'Invoice'; comment on column example.invoice_product.product_id is 'Product'; comment on column example.invoice_product.quantity is 'Quantity'; comment on column example.invoice_product.color is 'Color'; create index on example.invoice_product(invoice_id); -- Enable history call pghist.hist_enable('example', 'invoice'); call pghist.hist_enable('example', 'invoice_product', 'example', 'invoice'); -- Change data insert into example.invoice values (12,'#20', current_date, 1, 120.00); update example.invoice set customer_id=2 where id=12; insert into example.product(id,name,code) values (101,'Pensil','030'),(102,'Notebook','040'); insert into example.invoice_product(id, invoice_id, product_id, quantity, color) values (1,12,101,1000,'R'),(2,12,101,10,'G'); do $$ begin update example.invoice_product set quantity=quantity+1,color='B' where id=1; delete from example.invoice_product where id=2; update example.invoice set amount=150 where id=12; end; $$; -- Select all changes, first three columns provide chronological select * from example.invoice_changes() order by 1,2,3; -- Select all changes with column id (immutable), insert detail by columns, without detail table example.invoice_product select * from example.invoice_changes(hist_columns_insert=>true, hist_columns_immutable=>true, hist_tables_detail=>false) order by 1,2,3; -- Select partial changes by id with detail table, autocreated indexes provide fast execution select * from example.invoice_changes(12) order by 1,2,3; -- Equivalent with using union all select * from example.invoice_changes(id=>12, hist_tables_detail=>false) union all select * from example.invoice_product_changes(invoice_id=>12) order by 1,2,3; -- Set description expression for columns row_desc and value_desc call pghist.hist_expression_row_desc('example', 'invoice', '''Invoice'''); call pghist.hist_expression_row_desc('example', 'invoice_product', $$ 'Row #'||$1.id||' / '||(select name from example.product where id=$1.product_id) $$); call pghist.hist_expression_value_desc('example', 'invoice_product', 'color', $$ case when $1='R' then 'Red' when $1='B' then 'Blue' when $1='G' then 'Green' else $1 end $$); -- Replace function for column db_user_name create or replace function example.db_user_name(db_user name) returns varchar language plpgsql as $$ begin return '['||db_user||']'; end; $$; call pghist.hist_column_custom_function('db_user_name', 'example.db_user_name');
Changes to a specific condition can be obtained using a cross-join with a list of primary keys. To union history of multiple tables, function results can be combined using the union all operator. For such queries, it is recommended to create wrapper functions.
For data schemas with a complex structure, it is recommended to create: functions [schema].[table]_changes_ui, one hist_table_changes_ui data type and one webpage to display history of any table.
Example:
-- Create function that returns changes by date create or replace function example.invoice_changes_ui_date(date_changes date) returns setof pghist.hist_table_change language sql security definer as $$ select c.* from (select distinct id from example.invoice_hist where hist_timestamp::date=date_changes) h cross join example.invoice_changes(h.id) c order by 1,2,3; $$; select * from example.invoice_changes_ui_date(current_date); -- Create function that returns changes to master and detail tables create or replace function example.invoice_changes_ui_simple(id int) returns setof pghist.hist_table_change language sql security definer as $$ select * from example.invoice_changes(id=>id,hist_tables_detail=>false,hist_columns_insert=>true) union all select * from example.invoice_product_changes(invoice_id=>id) order by 1,2,3; $$; select * from example.invoice_changes_ui_simple(12); -- Create type with only necessary columns for the UI and cast function to it, -- type is created only once (as a rule, there is only one history view form per project) create type example.hist_table_change_ui as ( timestamp timestamptz, operation_name varchar, db_user_name varchar, row_desc text, column_comment varchar, value_old_desc text, value_new_desc text ); create or replace function example.hist_table_change_ui_cast(c pghist.hist_table_change) returns example.hist_table_change_ui language plpgsql as $$ begin return (c.timestamp,c.operation_name,c.db_user_name,c.row_desc,c.column_comment,c.value_old_desc,c.value_new_desc)::example.hist_table_change_ui; end; $$; create cast(pghist.hist_table_change as example.hist_table_change_ui) with function example.hist_table_change_ui_cast as assignment; -- Create function that returns only necessary columns create or replace function example.invoice_changes_ui(id int) returns setof example.hist_table_change_ui language sql security definer as $$ select c::pghist.hist_table_change::example.hist_table_change_ui from ( select * from example.invoice_changes(id=>id,hist_tables_detail=>false,hist_columns_insert=>true) union all select * from example.invoice_product_changes(invoice_id=>id) order by 1,2,3 ) c; $$; select * from example.invoice_changes_ui(12);
3. [schema].[table]_at_timestamp function creates copy of table (temporary table) at a point in time in the past and returns it
Used for data recovery and SQL queries from multiple tables at a point in time in the past, can be joined to the [schema].[table]_hist view by primary key.
For optimazation of SQL queries, the created temporary table has indexes similar to original, recovery time can be set once in the pghist.at_timestamp parameter.
Example:
-- Create schema and tables, enable history drop schema if exists example cascade; create schema example; create table example.customer( id int primary key, name varchar(100) not null ); insert into example.customer values (1,'Horns'),(2,'Hooves'); create table example.document( id int primary key, number varchar(10), date date ); create table example.invoice( primary key (id), customer_id int references example.customer(id), amount numeric(20,2) ) inherits (example.document); create index on example.invoice(customer_id); call pghist.hist_enable('example', 'customer'); call pghist.hist_enable('example', 'document'); call pghist.hist_enable('example', 'invoice'); -- Enter data into tables insert into example.document values (11,'#10', current_date); insert into example.invoice values (12,'#20', current_date, 1, 120.00); insert into example.invoice values (13,'#30', current_date-1, 2, 130.00); -- Select data in past select * from example.document_at_timestamp(now()-interval '10 second'); -- Erroneous update and recovery update example.invoice set customer_id=2,amount=300 where date=current_date; update example.invoice i set amount = h.amount from example.invoice_at_timestamp('2024-04-06 10:00:00') h where i.id = h.id and i.date=current_date; -- Сombination log and versioning select * from example.invoice_at_timestamp('2024-04-06 10:00:00') where id in ( select id from example.invoice_hist where hist_timestamp>'2024-04-06 10:00:00' and hist_db_user=current_user ); -- Сomplex query in past select * from example.invoice i join example.customer c on c.id=i.customer_id; do $$ begin perform set_config('pghist.at_timestamp', '2024-04-06 10:00:00', true); perform example.invoice_at_timestamp(); perform example.customer_at_timestamp(); end; $$; select * from example_invoice_at_timestamp i join example_customer_at_timestamp c on c.id=i.customer_id;
Security
History table hist_data$[schema]_[table] and trigger functions are created in pghist schema, objects for working with history are created in the table schema. Owner of all created objects is superuser, stored procedures are created with option security definer. Owner of table is granted privilegies with grant option to select from view [schema].[table]_hist and execute functions [schema].[table]_changes, [schema].[table]_at_timestamp.
If necessary, superuser grants select on common tables to developer.
Data access is shown in schema.
Example:
-- postgres grant privileges on pghist schema and pghist.hist_enable procedure to developer_1 -- all privileges in pghist_grants.sql file grant usage on schema pghist to developer_1; grant execute on procedure pghist.hist_enable(name) to developer_1; -- developer_1 create table, enable history, grant privileges on example table and example_changes function to user_1 create table example( id int primary key, name varchar(20), number numeric(10,2), date date ); call pghist.hist_enable('example'); grant select,insert,update on example to user_1; grant execute on function example_changes to user_1; -- user_1 change data and view changes insert into example values (1, 'Example', 10, current_date); update example set number=20, date=date-1; select * from example_changes() order by 1,2,3; -- When trying to select log, user_1 receives an error -- SQL Error [42501]: ERROR: permission denied for view example_hist select * from example_hist;
pghist.hist_enable(schema, table_name, master_table_schema, master_table_name, columns_excluded)
Procedure enable keeping history of changes for table, create history table, triggers and functions that return historical data
Parameters:
schema | name | - | table schema (optional) | ||
table_name | name | - | table name | ||
master_table_schema | name | - | master table schema (optional) | ||
master_table_name | name | - | master table name (optional) | ||
columns_excluded | name[] | - | excluded columns (optional) |
Examples:
call pghist.hist_enable('document'); call pghist.hist_enable('example', 'invoice'); call pghist.hist_enable('example', 'invoice_product', 'example', 'invoice'); call pghist.hist_enable('orm', 'myclass', columns_excluded => array['inserted_at','updated_at']);
[schema].[table]_hist
View contains a list of changes by row, created when history keeping is enabled.
Sorted by first two columns to provide chronology
Columns:
hist_statement_num | bigint | - | common SQL statement number | ||
hist_row_num | bigint | - | row number in dataset that modified SQL statement | ||
hist_timestamp | timestamptz | - | SQL statement start date-time | ||
hist_operation | varchar | - | operation: INSERT,UPDATE,DELETE,TRUNCATE | ||
hist_update_columns | name[] | - | changed columns for UPDATE operation | ||
hist_db_user | name | - | database user | ||
hist_app_user | varchar | - | application user defined as current_setting('app.user') | ||
hist_application_name | varchar | - | application name | ||
hist_query_text | text | - | SQL query text | ||
hist_statement_id | bigint | - | SQL statement ID referenced on pghist.hist_statement table | ||
[primary key column(s)] | - | primary key value (can be several) | |||
[column(s) of fkey on master table] | - | foreign key value on master table (optional, can be several) | |||
[column...]_old | - | old value for each column of table [schema].[table] |
Examples:
select * from example.document_hist; select * from example.document_hist where id=2; select * from example.document_hist where hist_operation='DELETE';
[schema].[table]_changes(hist_columns_immutable, hist_columns_insert, hist_tables_detail, hist_tables_inherited)
Function return changes in the table (setof pghist.table_change), created when history keeping is enabled.
Sorting by the first three columns provides chronology by time, to get related data from multiple tables, recommended to use operator union all
Parameters:
[primary key column(s)] | - | primary key value (optional, can be several) | |||
[column(s) of fkey on master table] | - | foreign key value on master table (optional, can be several) | |||
hist_columns_immutable | boolean | - | return values of immutable columns (optional, default false) | ||
hist_columns_insert | boolean | - | detail INSERT operation by column (optional, default false) | ||
hist_tables_detail | boolean | - | return changes in detail tables (optional, default true) | ||
hist_tables_inherited | boolean | - | process inherited tables (optional, default true) |
Returned dataset columns (type pghist.hist_table_change):
statement_num | bigint | - | common SQL statement number | ||
row_num | int | - | row number in dataset that modified SQL statement | ||
column_num | int | - | column number in row | ||
timestamp | timestamptz | - | operation date-time | ||
operation | varchar | - | operation: INSERT,UPDATE,DELETE,TRUNCATE | ||
operation_name | varchar | - | operation name, specified by the overridden function pghist.hist_custom_operation_name | ||
column_name | name | - | column name | ||
column_comment | varchar | - | column comment | ||
value_old | text | - | old value | ||
value_old_desc | text | - | description of old value, can be overridden by the procedure pghist.hist_expression_value_desc | ||
value_new | text | - | new value | ||
value_new_desc | text | - | description of new value, can be overridden by the procedure pghist.hist_expression_value_desc | ||
row_desc | text | - | description of row, can be overridden by the procedure pghist.hist_expression_row_desc | ||
db_user | name | - | database user | ||
db_user_name | varchar | - | database user name, specified by the overridden function pghist.hist_custom_db_user_name | ||
app_user | varchar | - | application user, specified by the overridden function pghist.hist_custom_app_user | ||
db_user_name | varchar | - | application user name, specified by the overridden function pghist.hist_custom_app_user_name | ||
schema | name | - | schema | ||
table_name | varchar | - | table name | ||
table_comment | varchar | - | table comment |
Examples:
select * from example.invoice_changes() order by 1,2,3; select * from example.invoice_changes(hist_columns_immutable=>true) order by 1,2,3; select * from example.invoice_changes(12) order by 1,2,3; select * from example.invoice_changes(id=>12, hist_columns_insert=>true, hist_tables_detail=>false) union all select * from example.invoice_product_changes(invoice_id=>12) order by 1,2,3;
[schema].[table]_at_timestamp(transaction_timestamp, cascade)
Function created a temporary table [schema]_[table]_at_timestamp as copy of the original table at point in time and returns it
Parameters:
schema | - | schema | |||
table_name | - | table name | |||
transaction_timestamp | timestamptz | - | date-time of transaction completion, (optional, default current_setting('pghist.at_timestamp')) | ||
cascade | boolean | - | process inherited tables (optional, default true) |
Examples:
select * from example.document_at_timestamp(now()-interval '1 hour'); do $$ begin perform set_config('pghist.at_timestamp', '2024-04-06 10:00:00', true); perform example.invoice_at_timestamp(); perform example.customer_at_timestamp(); end; $$; select * from example_invoice_at_timestamp i join example_customer_at_timestamp c on c.id=i.customer_id;
pghist.hist_disable(schema, table_name)
Procedure disable keeping history of changes for table, delete the necessary objects
Parameters:
schema | name | - | schema (optional) | ||
table_name | name | - | table name |
Examples:
call pghist.hist_disable('document'); call pghist.hist_disable('example', 'invoice');
SQL expressions of descriptions
pghist.hist_expression_row_desc(schema, table_name, expression)
pghist.hist_expression_value_desc(schema, table_name, column_name, expression)
The procedures set expression to describe row or value of field.
Variable $1 with row/field value is available in expression
pghist.hist_expression_row_desc_current(schema, table_name)
pghist.hist_expression_value_desc_current(schema, table_name, column_name)
Functions return current expression to describe row or value of field
pghist.hist_expression_row_desc_default(schema, table_name)
pghist.hist_expression_value_desc_default(schema, table_name, column_name)
Functions return default expression to describe row or value of field.
For row, table comment with the primary key is returned.
For column that is foreign key, first text field of foreigned table is returned.
Parameters:
schema | name | - | schema | ||
table | name | - | table name | ||
column_name | name | - | column name | ||
expression | varchar | - | expression, null discards the saved description |
Examples:
call pghist.hist_expression_row_desc('example', 'invoice', '''Invoice'''); call pghist.hist_expression_row_desc('example', 'invoice_product', $$ 'Row #'||$1.id||' / '||(select name from example.product where id=$1.product_id) $$); call pghist.hist_expression_value_desc('example', 'invoice_product', 'color', $$ case when $1='R' then 'Red' when $1='B' then 'Blue' when $1='G' then 'Green' else $1 end $$); select pghist.hist_expression_row_desc_current('example', 'invoice_product'); select pghist.hist_expression_value_desc_default('example', 'invoice_product', 'color');
Customization functions
To customize and localize fields in a specific application, can be define custom functions.
Recommended to use the default function in SQL manager (such as pgAdmin or DBeaver), override it and save in application schema as different name
Columns available for customization:
operation_name | varchar | - | operation name | ||
db_user_name | varchar | - | database user name | ||
app_user | varchar | - | application user | ||
app_user_name | varchar | - | application user name | ||
app_client_addr | inet | - | client application IP address | ||
app_client_hostname | varchar | - | client application host name |
pghist.hist_column_custom_function(column_name, custom_function)
The procedure sets function to get column value
Parameters:
column_name | name | - | column name | ||
custom_function | name | - | function name without parameters |
pghist.hist_column_custom_function(column_name)
The function returns current function to get column value
Parameter:
column_name | name | - | column name |
Example:
select pghist.hist_column_custom_function_current('db_user_name'); create or replace function myapp.user_name(db_user name) returns varchar language plpgsql as $$ begin return (select name from myapp.users where login=db_user); end; $$; call pghist.hist_column_custom_function('db_user_name', 'myapp.user_name');
History table pghist.hist_data$[schema]_[table]
History table is created in addition to the main table, developer (owner of main table) has access through the view [schema].[table]_hist
Columns:
hist_statement_id | bigint | - | SQL statement ID referenced on pghist.hist_statement table | ||
hist_row_num | int | - | row number in dataset that modified SQL statement | ||
hist_update_columns | name[] | - | changed columns for UPDATE operation | ||
[primary key column(s)] | - | primary key value (can be several) | |||
[column(s) of fkey on master table] | - | foreign key value on master table (optional, can be several) | |||
[column...]_old | - | old value for each column of table [schema].[table] |
Example:
select * from pghist.hist_data$example_invoice h join pghist.hist_statement s on s.id=h.hist_statement_id join pghist.hist_query q on q.hash=s.query_hash where q.text like 'do%begin%';
Common tables
pghist.hist_transaction | - | transactions |
pghist.hist_query | - | SQL queries |
pghist.hist_statement | - | SQL statements |
pghist.hist_table | - | history tables |
pghist.hist_table_column | - | columns of history tables |
pghist.hist_column_custom_function | - | custom functions of columns |
On installation created tables that store common data (transaction date-time and SQL expressions, operation, SQL query, user, session, etc) and tables with settings. Modification of common tables is allowed using stored procedures. By default, developers (owners of main tables) do not have access to common tables; if necessary, privilegies are granted manually.
History table pghist.hist_data$[schema]_[table] by field statement_id refers to pghist.hist_statement, which in, in turn, refers to pghist.hist_transaction and pghist.hist_query.
Example:
select * from pghist.hist_statement s join pghist.hist_transaction t on t.id=s.transaction_id join pghist.hist_query q on q.hash=s.query_hash where t.timestamp_commit between '2024-04-06 10:00:00' and '2024-04-06 11:00:00'; select * from pghist.hist_table ht left join pghist.hist_table_column htc on htc.schema=ht.schema and htc.table_name=ht.name;
pghist.pghist_version
The function returns tool version
pghist.hist_sql_log
All executed commands logging in table pghist.hist_sql_log
Examples:
select * from pghist.hist_sql_log where schema='example' and table_name='document' order by id; select * from pghist.hist_sql_log l join pghist.hist_transaction t on t.id=l.transaction_id where schema='example' and table_name='document' order by 1 desc;