This is a concept on how to run the Logidze gem with a scope.
As it is, logidze uses an at() method to retrieve historical versions. This must be applied to each record in each table individually, like record.at(time: 2.days.ago).
But in practice we are not much concerned with single record history. Usually we have a fully normalized database structure with lots of tables (that is, lots of models) all connected together via foreign keys to form a hierarchical tree. Ad we want to access historical versions of the entire tree (or specific subsets from it).
Logidze has a feature to handle associations - but the docs say it suffers from some corner cases, so I tend to doubt if this would work well with a few dozen of tables which are zig-zag related in multiple ways by lots of foreign keys (which is how a properly normalized db structure does look like).
Anyway, it will not work with e.g. deep_clone, because there is no way to tell deep_clone to correctly apply the at() method.
If, instead, the at() method would work as a scope, then we could simply attach that scope to ApplicationRecord, and then the entire database content would time-travel as a whole, and the association problem would disappear into thin air.
Furthermore, scopes are automatically joined, so any other scope (like current_user) would just blend in without extra care.
The issue with this is: scopes are computed within the database itself. After being constructed, they are given to Arel which produces pure SQL. So, since the at() method is ruby code, it cannot be part of a scope. What we need to do is, rewrite the at() method in Pl/SQL.
I scripted a simple version of this, a historical() function in Pl/SQL. It does only accept the native timestamps of Logidze as a parameter, and delivers the historical version of a record as json. But this is enough to get things flying.
has_logidze statements from your models and instead put the statement into ApplicationRecord (or some appropriate intermediate superclass).historical() function as follows, and run it.class LogidzeHistInstall < ActiveRecord::Migration[6.1]
def up
execute <<-SQL
CREATE OR REPLACE FUNCTION historical(
log_data jsonb,
timestp bigint)
RETURNS jsonb
LANGUAGE 'plpgsql'
COST 100
VOLATILE PARALLEL UNSAFE
AS $BODY$
DECLARE
histj jsonb;
BEGIN
-- missing log data:
IF (log_data#>'{h,0,ts}')::bigint IS NULL OR
log_data#>'{h,0,v}' IS NULL OR
log_data->'v' IS NULL THEN
return NULL;
END IF;
-- record not yet present at given time:
IF (log_data#>'{h,0,ts}')::bigint > timestp THEN
return NULL;
END IF;
histj = log_data#>'{h,0,c}';
LOOP
EXIT WHEN log_data#>'{h,0,v}' = log_data->'v';
EXIT WHEN (log_data#>'{h,1,ts}')::bigint > timestp;
histj := histj || (log_data#>'{h,1,c}');
log_data := jsonb_set(log_data, '{h}', (log_data->'h') - 0);
END LOOP;
return histj;
END;
$BODY$;
SQL
end
def down
execute <<-SQL
DROP FUNCTION historical(jsonb, bigint);
SQL
end
endclass ApplicationRecord < ActiveRecord::Base
self.abstract_class = true
include Discard::Model
has_logidze
scope :hist, ->(timepoint) {
thisrel = self.name.downcase.pluralize
tbalias = "hcur"
histqf = "historical(#{tbalias}.log_data, #{timepoint})"
jsoncf = "jsonb_populate_record(null::#{thisrel}, #{histqf})"
from("#{thisrel} as #{tbalias}, #{jsoncf} as #{thisrel}")
}
...
With this we can now retrieve historical versions from our models:Model.all.hist(1628283028010)
This appears to be no big achievement, because we could do the same with the at() method:Model.all.at(time: 1628283028010)
The fun starts when we add further clauses:Model.all.hist(1628283028010).where(id: 11)
This does work.
Model.all.at(time: 1628283028010).where(id: 11)
This does not work.
So why do we need this? We could make the at() method always be the last one in the query, and that would also work. But, there is a difference if we apply conditions to a query and then go back in history for only the records found, or if we go back in history for all records and then apply our condition to the historical data.
Furthermore, this scope can now be made to work with deep_clone.
Named scopes in rails do apply only to a certain model. They are supposed to somehow qualify a collection that is retrieved from a model. And if we would want to have the same scope applied to some association attached to the model, we would need to apply the scope to that association again:
Post.hist(1628303790000).first.comments.hist(1628303790000)
Post Load (1.9ms) SELECT "posts".* FROM posts as hcur, jsonb_populate_record(null::posts, historical(hcur.log_data, 1628303790000)) as posts WHERE "posts"."discarded_at" IS NULL AND "posts"."id" IS NOT NULL ORDER BY "posts"."id" ASC LIMIT $1 [["LIMIT", 1]]
Comment Load (1.5ms) SELECT "comments".* FROM comments as hcur, jsonb_populate_record(null::comments, historical(hcur.log_data, 1628303790000)) as comments WHERE "comments"."discarded_at" IS NULL AND "comments"."post_id" = $1 AND "comments"."id" IS NOT NULL /* loading for inspect */ LIMIT $2 [["post_id", 11], ["LIMIT", 11]]Again, we could do mostly the same with the at() method:
Post.at(time: 1628303790000).first.comments.at(time: 1628303790000)
Post Load (1.4ms) SELECT "posts".* FROM "posts" WHERE "posts"."discarded_at" IS NULL
Comment Load (1.0ms) SELECT "comments".* FROM "comments" WHERE "comments"."discarded_at" IS NULL AND "comments"."post_id" = $1 [["ruleset_id", 17]][We can see that the sort order is now different: the first clause, being applied behind the at() method, is not propagated to the database.]
I do not know if there is a way in Rails to automatically apply the same named scope to all dependent associations in a query. But there certainly is a default scope. And if that is declared in the superclass (ApplicationRecord or something more appropriate), then it will apply to all dependent associations.
The use of the default_scope is generally discouraged, because it might apply in cases where we do not want it to apply, and then give weird results. But here this is what is needed.
Just like any scope, the default scope can have parameters (like the hist() scope here has a timestamp as the parameter), and -more important- these parameters can be changed inflight, at any point. Obviousely this makes the default scope even more dangerous than it is by nature, and we must take good care that such changing of the parameters is thread-safe, i.e. it will apply only to the currently running thread, and that we reset it to a harmless value before doing anything else.
A simple implementation of this might look as follows:
class ApplicationRecord < ActiveRecord::Base
...
@@historical = Concurrent::ThreadLocalVar.new(0)
@@showdiscarded = Concurrent::ThreadLocalVar.new(false)
def self.historical timepoint, showdiscarded
@@historical.value = timepoint
@@showdiscarded.value = showdiscarded
end
default_scope {
dfltscope = @@showdiscarded.value ? :all : :kept
if @@historical.value == 0
send(dfltscope)
else
hist(@@historical.value).send(dfltscope)
end
}
endAnd with that now our deep_clone can work, and the entire database can fly in time:
...
timestmp = params[:hist].to_i
ApplicationRecord.historical(timestmp, true)
@post = Post.find(params[:id])
post = @post.deep_clone include: HIERARCHY, use_dictionary: true
ApplicationRecord.historical(0, false)
post.save
...The actual application for which I need this to work, is systems configurations databases: they contain parameters used to install unix machines, like network routing, application configuration, firewall rules, etc.etc. - many many tables all somehow related to another.
And there is a need to answer questions like "how did all this look last week when it still did work?" or "lets rollback to the exact configuration as it was before the change".