Usage method


Call of pghist.hist_enable procedure creates 3 objects in the table schema for working with history:

1. [schema].[table]_hist view is a change log (audit) for developers to analyze data problems

SQL query and application name allow to determine the location in the source code, transaction identifier - changes in other tables and SQL queries.

Example:
-- Create schema and table, enable history, change data 
drop schema if exists example cascade;
create schema example;

create table example.document(
  id int primary key,
  number varchar(10),
  amount numeric(10,2)
);

call pghist.hist_enable('example', 'document');

insert into example.document values (11, '#10', 100);
insert into example.document values (12, '#20', 200);
update example.document set number='#20/2',amount=210 where id=12;
update example.document set amount=220 where id=12;
delete from example.document where id=11;

-- Select full info
select * from example.document_hist;

-- Select partial info by id
select hist_timestamp,hist_operation,hist_db_user,amount_old 
  from example.document_hist
  where id=12 and (hist_operation!='UPDATE' or 'amount'=any(hist_update_columns))
  order by hist_statement_id;

-- Select partial info with current values
select * from example.document_hist
union all 
select null,null,null,'[CURRENT_VALUES]',null,null,null,null,null,null,* from example.document
order by id,1;

-- Create extended view and grant select on it to developer
-- (alternatively to 'grant select on all tables in schema pghist to developer');  
select *
  from pghist.hist_data$example_document h  
  join pghist.hist_statement s on s.id = h.hist_statement_id
  join pghist.hist_query q on q.hash = s.query_hash
  join pghist.hist_transaction t on t.id = s.transaction_id
  order by h.hist_statement_id, h.id;
 
create or replace view example.document_hist_ext as  
  select h.id,t.timestamp_commit,t.db_client_addr,q.text query_text
    from pghist.hist_data$example_document h  
    join pghist.hist_statement s on s.id = h.hist_statement_id
    join pghist.hist_query q on q.hash = s.query_hash
    join pghist.hist_transaction t on t.id = s.transaction_id
    order by h.hist_statement_id, h.id;
   
select * from example.document_hist_ext where id=12;   

grant select on example.document_hist_ext to developer_1;      


2. [schema].[table]_changes function returns a set of data to display changes to the user

Information is displayed in the old-new value format with a description.

Display identifiers and foreign keys to the user is uninformative, so for each row and field values description is displayed that can be overridden via SQL expression. By default, comment table with the primary key is returned for a row description, first text field of related table is returned for a column that is the foreign key, for others - value. Thus, can display to user a description without value.

When enabling history for detail table, must specify the master table, foreign key values will be stored and indexed. By default, when changes are received in the master table, changes in the detail tables will also be returned.

Some columns (for example, username) can be redefined via customization functions.

Example:
-- Create schema and tables
drop schema if exists example cascade;
create schema example;

create table example.customer(
  id int primary key,
  name varchar(100) not null
);
insert into example.customer values (1,'Horns'),(2,'Hooves');

create table example.invoice(
  id int primary key,  
  number varchar(10),
  date date,  
  customer_id int references example.customer(id),
  amount numeric(20,2) 
);
comment on table example.invoice is 'Invoice';
comment on column example.invoice.id is 'Identifier';
comment on column example.invoice.number is 'Number';
comment on column example.invoice.date is 'Date';
comment on column example.invoice.customer_id is 'Сustomer';
comment on column example.invoice.amount is 'Amount';

create table example.product(
  id int primary key,
  name varchar(100) not null,
  code varchar(10) not null
);

create table example.invoice_product(
  id serial primary key,
  invoice_id int references example.invoice(id),   
  product_id int references example.product(id),  
  quantity int,
  color char(1) check (color in ('R','G','B'))  
);
comment on table example.invoice_product is 'Product of invoice';
comment on column example.invoice_product.id is 'Identifier';
comment on column example.invoice_product.invoice_id is 'Invoice';
comment on column example.invoice_product.product_id is 'Product';
comment on column example.invoice_product.quantity is 'Quantity';
comment on column example.invoice_product.color is 'Color';
create index on example.invoice_product(invoice_id);

-- Enable history
call pghist.hist_enable('example', 'invoice');
call pghist.hist_enable('example', 'invoice_product', 'example', 'invoice');

-- Change data
insert into example.invoice values (12,'#20', current_date, 1, 120.00);
update example.invoice set customer_id=2 where id=12;
insert into example.product(id,name,code) values (101,'Pensil','030'),(102,'Notebook','040');
insert into example.invoice_product(id, invoice_id, product_id, quantity, color) values (1,12,101,1000,'R'),(2,12,101,10,'G');

do $$
begin
  update example.invoice_product set quantity=quantity+1,color='B' where id=1;
  delete from example.invoice_product where id=2;
  update example.invoice set amount=150 where id=12;
end; 
$$;

-- Select all changes, first three columns provide chronological
select * from example.invoice_changes() order by 1,2,3;

-- Select all changes with column id (immutable), insert detail by columns, without detail table example.invoice_product
select * from example.invoice_changes(hist_columns_insert=>true, hist_columns_immutable=>true, hist_tables_detail=>false) order by 1,2,3; 

-- Select partial changes by id with detail table, autocreated indexes provide fast execution
select * from example.invoice_changes(12) order by 1,2,3;

-- Equivalent with using union all  
select * from example.invoice_changes(id=>12, hist_tables_detail=>false)
union all
select * from example.invoice_product_changes(invoice_id=>12)
order by 1,2,3;

-- Set description expression for columns row_desc and value_desc   
call pghist.hist_expression_row_desc('example', 'invoice', '''Invoice''');   
call pghist.hist_expression_row_desc('example', 'invoice_product', $$ 'Row #'||$1.id||' / '||(select name from example.product where id=$1.product_id) $$);
call pghist.hist_expression_value_desc('example', 'invoice_product', 'color', $$ case when $1='R' then 'Red' when $1='B' then 'Blue' when $1='G' then 'Green' else $1 end $$);

-- Replace function for column db_user_name
create or replace function example.db_user_name(db_user name) returns varchar language plpgsql as $$ 
begin 
  return '['||db_user||']';                                        
end; $$;
call pghist.hist_column_custom_function('db_user_name', 'example.db_user_name');

Changes to a specific condition can be obtained using a cross-join with a list of primary keys. To union history of multiple tables, function results can be combined using the union all operator. For such queries, it is recommended to create wrapper functions.

For data schemas with a complex structure, it is recommended to create: functions [schema].[table]_changes_ui, one hist_table_changes_ui data type and one webpage to display history of any table.

Example:
-- Create function that returns changes by date
create or replace function example.invoice_changes_ui_date(date_changes date) returns setof pghist.hist_table_change language sql security definer as $$
  select c.*
    from (select distinct id from example.invoice_hist where hist_timestamp::date=date_changes) h
    cross join example.invoice_changes(h.id) c
  order by 1,2,3;
$$;
select * from example.invoice_changes_ui_date(current_date);

-- Create function that returns changes to master and detail tables
create or replace function example.invoice_changes_ui_simple(id int) returns setof pghist.hist_table_change language sql security definer as $$
  select * from example.invoice_changes(id=>id,hist_tables_detail=>false,hist_columns_insert=>true)
  union all
  select * from example.invoice_product_changes(invoice_id=>id)
  order by 1,2,3;
$$;
select * from example.invoice_changes_ui_simple(12);

-- Create type with only necessary columns for the UI and cast function to it,
-- type is created only once (as a rule, there is only one history view form per project) 
create type example.hist_table_change_ui as (
  timestamp timestamptz,
  operation_name varchar,
  db_user_name varchar,
  row_desc text,  
  column_comment varchar,      
  value_old_desc text,      
  value_new_desc text
);
create or replace function example.hist_table_change_ui_cast(c pghist.hist_table_change) returns example.hist_table_change_ui language plpgsql as $$
begin
  return (c.timestamp,c.operation_name,c.db_user_name,c.row_desc,c.column_comment,c.value_old_desc,c.value_new_desc)::example.hist_table_change_ui; 
end; $$;
create cast(pghist.hist_table_change as example.hist_table_change_ui) with function example.hist_table_change_ui_cast as assignment;

-- Create function that returns only necessary columns
create or replace function example.invoice_changes_ui(id int) returns setof example.hist_table_change_ui language sql security definer as $$
  select c::pghist.hist_table_change::example.hist_table_change_ui from ( 
    select * from example.invoice_changes(id=>id,hist_tables_detail=>false,hist_columns_insert=>true)
    union all
    select * from example.invoice_product_changes(invoice_id=>id)
    order by 1,2,3
  ) c;  
$$;
select * from example.invoice_changes_ui(12);


3. [schema].[table]_at_timestamp function creates copy of table (temporary table) at a point in time in the past and returns it

Used for data recovery and SQL queries from multiple tables at a point in time in the past, can be joined to the [schema].[table]_hist view by primary key.

For optimazation of SQL queries, the created temporary table has indexes similar to original, recovery time can be set once in the pghist.at_timestamp parameter.

Example:
--  Create schema and tables, enable history
drop schema if exists example cascade;
create schema example;

create table example.customer(
  id int primary key,
  name varchar(100) not null
);
insert into example.customer values (1,'Horns'),(2,'Hooves');

create table example.document(
  id int primary key,
  number varchar(10),
  date date
);

create table example.invoice(
  primary key (id),
  customer_id int references example.customer(id),
  amount numeric(20,2)
) inherits (example.document);
create index on example.invoice(customer_id);

call pghist.hist_enable('example', 'customer');
call pghist.hist_enable('example', 'document');
call pghist.hist_enable('example', 'invoice');

-- Enter data into tables   
insert into example.document values (11,'#10', current_date);
insert into example.invoice  values (12,'#20', current_date,   1, 120.00);
insert into example.invoice  values (13,'#30', current_date-1, 2, 130.00);

-- Select data in past
select * from example.document_at_timestamp(now()-interval '10 second');

-- Erroneous update and recovery
update example.invoice set customer_id=2,amount=300 where date=current_date;

update example.invoice i
    set amount = h.amount
  from example.invoice_at_timestamp('2024-04-06 10:00:00') h
  where i.id = h.id
    and i.date=current_date;

-- Сombination log and versioning   
select * 
  from example.invoice_at_timestamp('2024-04-06 10:00:00')
  where id in (
          select id from example.invoice_hist
            where hist_timestamp>'2024-04-06 10:00:00'
              and hist_db_user=current_user
        );

-- Сomplex query in past        
select * 
  from example.invoice i
  join example.customer c on c.id=i.customer_id;
 
do $$
begin
  perform set_config('pghist.at_timestamp', '2024-04-06 10:00:00', true);    
  perform example.invoice_at_timestamp();    
  perform example.customer_at_timestamp();
end; 
$$;
 
select * 
  from example_invoice_at_timestamp i
  join example_customer_at_timestamp c on c.id=i.customer_id;

Security


History table hist_data$[schema]_[table] and trigger functions are created in pghist schema, objects for working with history are created in the table schema. Owner of all created objects is superuser, stored procedures are created with option security definer. Owner of table is granted privilegies with grant option to select from view [schema].[table]_hist and execute functions [schema].[table]_changes, [schema].[table]_at_timestamp.

If necessary, superuser grants select on common tables to developer.

Data access is shown in schema.

Example:
-- postgres grant privileges on pghist schema and pghist.hist_enable procedure to developer_1
-- all privileges in pghist_grants.sql file
grant usage on schema pghist to developer_1;
grant execute on procedure pghist.hist_enable(name) to developer_1;


-- developer_1 create table, enable history, grant privileges on example table and example_changes function to user_1
create table example(
  id int primary key,
  name varchar(20),
  number numeric(10,2),
  date date
);

call pghist.hist_enable('example');
  
grant select,insert,update on example to user_1;
grant execute on function example_changes to user_1;


-- user_1 change data and view changes 
insert into example values (1, 'Example', 10, current_date);
update example set number=20, date=date-1;

select * from example_changes() order by 1,2,3;

-- When trying to select log, user_1 receives an error
-- SQL Error [42501]: ERROR: permission denied for view example_hist 
select * from example_hist;

pghist.hist_enable(schema, table_name, master_table_schema, master_table_name, columns_excluded)


Procedure enable keeping history of changes for table, create history table, triggers and functions that return historical data

Parameters:
schemaname-table schema (optional)
table_namename-table name
master_table_schemaname-master table schema (optional)
master_table_namename-master table name (optional)
columns_excludedname[]-excluded columns (optional)

Examples:
call pghist.hist_enable('document');

call pghist.hist_enable('example', 'invoice');
call pghist.hist_enable('example', 'invoice_product', 'example', 'invoice');

call pghist.hist_enable('orm', 'myclass', columns_excluded => array['inserted_at','updated_at']);

[schema].[table]_hist


View contains a list of changes by row, created when history keeping is enabled.
Sorted by first two columns to provide chronology

Columns:
hist_statement_numbigint-common SQL statement number
hist_row_numbigint-row number in dataset that modified SQL statement
hist_timestamptimestamptz-SQL statement start date-time
hist_operationvarchar-operation: INSERT,UPDATE,DELETE,TRUNCATE
hist_update_columnsname[]-changed columns for UPDATE operation
hist_db_username-database user
hist_app_uservarchar-application user defined as current_setting('app.user')
hist_application_namevarchar-application name
hist_query_texttext-SQL query text
hist_statement_idbigint-SQL statement ID referenced on pghist.hist_statement table
[primary key column(s)]-primary key value (can be several)
[column(s) of fkey on master table]-foreign key value on master table (optional, can be several)
[column...]_old-old value for each column of table [schema].[table]

Examples:
select * from example.document_hist;
select * from example.document_hist where id=2;
select * from example.document_hist where hist_operation='DELETE';

[schema].[table]_changes(hist_columns_immutable, hist_columns_insert, hist_tables_detail, hist_tables_inherited)


Function return changes in the table (setof pghist.table_change), created when history keeping is enabled.
Sorting by the first three columns provides chronology by time, to get related data from multiple tables, recommended to use operator union all

Parameters:
[primary key column(s)]-primary key value (optional, can be several)
[column(s) of fkey on master table]-foreign key value on master table (optional, can be several)
hist_columns_immutableboolean-return values of immutable columns (optional, default false)
hist_columns_insertboolean-detail INSERT operation by column (optional, default false)
hist_tables_detailboolean-return changes in detail tables (optional, default true)
hist_tables_inheritedboolean-process inherited tables (optional, default true)

Returned dataset columns (type pghist.hist_table_change):
statement_numbigint-common SQL statement number
row_numint-row number in dataset that modified SQL statement
column_numint-column number in row
timestamptimestamptz-operation date-time
operationvarchar-operation: INSERT,UPDATE,DELETE,TRUNCATE
operation_namevarchar-operation name, specified by the overridden function pghist.hist_custom_operation_name
column_namename-column name
column_commentvarchar-column comment
value_oldtext-old value
value_old_desctext-description of old value, can be overridden by the procedure pghist.hist_expression_value_desc
value_newtext-new value
value_new_desctext-description of new value, can be overridden by the procedure pghist.hist_expression_value_desc
row_desctext-description of row, can be overridden by the procedure pghist.hist_expression_row_desc
db_username-database user
db_user_namevarchar-database user name, specified by the overridden function pghist.hist_custom_db_user_name
app_uservarchar-application user, specified by the overridden function pghist.hist_custom_app_user
db_user_namevarchar-application user name, specified by the overridden function pghist.hist_custom_app_user_name
schemaname-schema
table_namevarchar-table name
table_commentvarchar-table comment

Examples:
select * from example.invoice_changes() order by 1,2,3;
select * from example.invoice_changes(hist_columns_immutable=>true) order by 1,2,3; 

select * from example.invoice_changes(12) order by 1,2,3;

select * from example.invoice_changes(id=>12, hist_columns_insert=>true, hist_tables_detail=>false)
union all
select * from example.invoice_product_changes(invoice_id=>12)
order by 1,2,3;

[schema].[table]_at_timestamp(transaction_timestamp, cascade)


Function created a temporary table [schema]_[table]_at_timestamp as copy of the original table at point in time and returns it

Parameters:
schema-schema
table_name-table name
transaction_timestamptimestamptz-date-time of transaction completion, (optional, default current_setting('pghist.at_timestamp'))
cascadeboolean-process inherited tables (optional, default true)

Examples:
select * from example.document_at_timestamp(now()-interval '1 hour');

do $$
begin
  perform set_config('pghist.at_timestamp', '2024-04-06 10:00:00', true);    
  perform example.invoice_at_timestamp();    
  perform example.customer_at_timestamp();
end; 
$$;
select * 
  from example_invoice_at_timestamp i
  join example_customer_at_timestamp c on c.id=i.customer_id;

pghist.hist_disable(schema, table_name)


Procedure disable keeping history of changes for table, delete the necessary objects

Parameters:
schemaname-schema (optional)
table_namename-table name

Examples:
call pghist.hist_disable('document');
call pghist.hist_disable('example', 'invoice');

SQL expressions of descriptions


pghist.hist_expression_row_desc(schema, table_name, expression)
pghist.hist_expression_value_desc(schema, table_name, column_name, expression)

The procedures set expression to describe row or value of field.
Variable $1 with row/field value is available in expression


pghist.hist_expression_row_desc_current(schema, table_name)
pghist.hist_expression_value_desc_current(schema, table_name, column_name)

Functions return current expression to describe row or value of field


pghist.hist_expression_row_desc_default(schema, table_name)
pghist.hist_expression_value_desc_default(schema, table_name, column_name)

Functions return default expression to describe row or value of field.
For row, table comment with the primary key is returned.
For column that is foreign key, first text field of foreigned table is returned.


Parameters:
schemaname-schema
tablename-table name
column_namename-column name
expressionvarchar-expression, null discards the saved description

Examples:
call pghist.hist_expression_row_desc('example', 'invoice', '''Invoice''');   
call pghist.hist_expression_row_desc('example', 'invoice_product', $$ 'Row #'||$1.id||' / '||(select name from example.product where id=$1.product_id) $$);
call pghist.hist_expression_value_desc('example', 'invoice_product', 'color', $$ case when $1='R' then 'Red' when $1='B' then 'Blue' when $1='G' then 'Green' else $1 end $$);

select pghist.hist_expression_row_desc_current('example', 'invoice_product');
select pghist.hist_expression_value_desc_default('example', 'invoice_product', 'color');

Customization functions


To customize and localize fields in a specific application, can be define custom functions.
Recommended to use the default function in SQL manager (such as pgAdmin or DBeaver), override it and save in application schema as different name

Columns available for customization:
operation_namevarchar-operation name
db_user_namevarchar-database user name
app_uservarchar-application user
app_user_namevarchar-application user name
app_client_addrinet-client application IP address
app_client_hostnamevarchar-client application host name

pghist.hist_column_custom_function(column_name, custom_function)


The procedure sets function to get column value

Parameters:
column_namename-column name
custom_functionname-function name without parameters

pghist.hist_column_custom_function(column_name)


The function returns current function to get column value

Parameter:
column_namename-column name

Example:
select pghist.hist_column_custom_function_current('db_user_name');

create or replace function myapp.user_name(db_user name) returns varchar language plpgsql as $$ 
begin 
  return (select name from myapp.users where login=db_user);                                        
end; $$;

call pghist.hist_column_custom_function('db_user_name', 'myapp.user_name');

History table pghist.hist_data$[schema]_[table]


History table is created in addition to the main table, developer (owner of main table) has access through the view [schema].[table]_hist

Columns:
hist_statement_idbigint-SQL statement ID referenced on pghist.hist_statement table
hist_row_numint-row number in dataset that modified SQL statement
hist_update_columnsname[]-changed columns for UPDATE operation
[primary key column(s)]-primary key value (can be several)
[column(s) of fkey on master table]-foreign key value on master table (optional, can be several)
[column...]_old-old value for each column of table [schema].[table]

Example:
select *
  from pghist.hist_data$example_invoice h
  join pghist.hist_statement s on s.id=h.hist_statement_id
  join pghist.hist_query q on q.hash=s.query_hash
  where q.text like 'do%begin%';

Common tables


pghist.hist_transaction-transactions
pghist.hist_query-SQL queries
pghist.hist_statement-SQL statements
pghist.hist_table-history tables
pghist.hist_table_column-columns of history tables
pghist.hist_column_custom_function-custom functions of columns

On installation created tables that store common data (transaction date-time and SQL expressions, operation, SQL query, user, session, etc) and tables with settings. Modification of common tables is allowed using stored procedures. By default, developers (owners of main tables) do not have access to common tables; if necessary, privilegies are granted manually.

History table pghist.hist_data$[schema]_[table] by field statement_id refers to pghist.hist_statement, which in, in turn, refers to pghist.hist_transaction and pghist.hist_query.

Example:
select *
  from pghist.hist_statement s
  join pghist.hist_transaction t on t.id=s.transaction_id
  join pghist.hist_query q on q.hash=s.query_hash
  where t.timestamp_commit between '2024-04-06 10:00:00' and '2024-04-06 11:00:00';

select *
  from pghist.hist_table ht
  left join pghist.hist_table_column htc on htc.schema=ht.schema and htc.table_name=ht.name;

pghist.pghist_version


The function returns tool version

pghist.hist_sql_log


All executed commands logging in table pghist.hist_sql_log

Examples:
select * from pghist.hist_sql_log where schema='example' and table_name='document' order by id;
select * from pghist.hist_sql_log l
  join pghist.hist_transaction t on t.id=l.transaction_id
  where schema='example' and table_name='document'
  order by 1 desc;