This library allows to preserve structural sharing of immer
containers while serializing and deserializing them.
Structural sharing allows immer
containers to be efficient. At runtime, two distinct containers can be operated on
independently but internally they share nodes and use memory efficiently in that way. However when such containers are
serialized in a simple direct way, for example, as lists, this sharing is lost: they become truly independent, same data
is stored multiple times on disk and later, when it is read from disk, in memory.
This library operates on the internal structure of immer
containers: allowing it to be serialized, deserialized and
transformed. This enables more efficient storage, particularly when many nodes are reused, and, even more importantly,
preserving structural sharing after deserializing the containers.
Consider this scenario: an application has a document type that internally uses an immer
container in multiple
places, for example, an immer::vector<std::string>
. Some of these vectors would be completely identical, while
others would have just a few elements different (stored in an undo history, for example). The goal is to apply a
transformation function to these vectors.
A direct approach would be to take each vector and create a new vector by applying the transformation function for each element. However, after this process, all the structural sharing of the original containers would be lost: the result would be multiple independent vectors without any structural sharing.
This library enables the application of the transformation function directly on the nodes, preserving structural sharing. Additionally, regardless of how many times a node is reused, the transformation needs to be performed only once.
In addition to the dependencies of immer
, this library makes use of C++20,
Boost.Hana, fmt and cereal.
For this example, we’ll use a document type that contains two immer
vectors.
// Set the BL constant to 1, so that only 2 elements are stored in leaves.
// This allows to demonstrate structural sharing even in vectors with just a few
// elements.
using vector_one =
immer::vector<int, immer::default_memory_policy, immer::default_bits, 1>;
struct document
{
vector_one ints;
vector_one ints2;
friend bool operator==(const document&, const document&) = default;
// Make the struct serializable with cereal as usual, nothing special
// related to immer-persist.
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(ints), CEREAL_NVP(ints2));
}
};
using json_t = nlohmann::json;
Let’s make the document
struct compatible with boost::hana
. This way, the persist
library can determine what
pool types are needed and to name the pools.
BOOST_HANA_ADAPT_STRUCT(document, ints, ints2);
Let’s say we have two vectors v1
and v2
, where v2
is derived from v1
so that it shares data with it:
const auto v1 = vector_one{1, 2, 3};
const auto v2 = v1.push_back(4).push_back(5).push_back(6);
const auto value = document{v1, v2};
We can serialize the document using cereal
with this:
auto os = std::ostringstream{};
{
auto ar = cereal::JSONOutputArchive{os};
ar(value);
}
return os.str();
Generating a JSON like this one:
{"value0": {"ints": [1, 2, 3], "ints2": [1, 2, 3, 4, 5, 6]}}
As you can see, ints
and ints2
contain the full linearization of each vector.
The structural sharing between these two data structures is not represented in its
serialized form. However, with immer-persist
we can serialize it with:
const auto policy =
immer::persist::hana_struct_auto_member_name_policy(document{});
const auto str = immer::persist::cereal_save_with_pools(value, policy);
Which generates some JSON like this:
const auto expected_json = json_t::parse(R"(
{
"value0": {"ints": 0, "ints2": 1},
"pools": {
"ints": {
"B": 5,
"BL": 1,
"inners": [
[0, {"children": [2], "relaxed": false}],
[3, {"children": [2, 5], "relaxed": false}]
],
"leaves": [[1, [3]], [2, [1, 2]], [4, [5, 6]], [5, [3, 4]]],
"vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
}
}
}
)");
As you can see, the value is serialized with every immer
container replaced by an identifier.
This identifier is a key into a pool, which is serialized just after.
A pool represents a set of immer
containers of a specific type. For example, we may have a pool that contains all
immer::vector<int>
of our document. You can think of it as a small database of immer
containers. When
serializing the pool, the internal structure of all those immer
containers is written, preserving the structural
sharing between those containers. The nodes of the trees that make up the immer
containers are directly represented
in the JSON and, because we are representing all the containers as a whole, those nodes that are referenced in
multiple trees can be stored only once. That same structure is preserved when reading the pool back from disk and
reconstructing the vectors (and other containers) from it, thus allowing us to preserve the structural sharing across
sessions.
Note
Currently, immer-persist
makes a distiction between pools used for saving containers (output pools) and for loading containers (input pools),
similar to cereal
with its InputArchive
and OutputArchive
distiction.
Currently, immer-persist
focuses on JSON as the serialization format and uses the cereal
library internally. In principle, other formats
and serialization libraries could be supported in the future.
We can use policy to control the names of the pools for each container.
For this example, let’s define a new document type doc_2
. It will also contain another type extra_data
with a
vector
of strings
in it. To demonstrate the responsibilities of the policy, the doc_2
type will not be a
boost::hana::Struct
and will not allow for compile-time reflection.
using vector_str = immer::
vector<std::string, immer::default_memory_policy, immer::default_bits, 1>;
struct extra_data
{
vector_str comments;
friend bool operator==(const extra_data&, const extra_data&) = default;
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(comments));
}
};
struct doc_2
{
vector_one ints;
vector_one ints2;
vector_str strings;
extra_data extra;
friend bool operator==(const doc_2&, const doc_2&) = default;
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(ints),
CEREAL_NVP(ints2),
CEREAL_NVP(strings),
CEREAL_NVP(extra));
}
};
We define the doc_2_policy
as following:
struct doc_2_policy
{
auto get_pool_types(const auto&) const
{
return boost::hana::tuple_t<vector_one, vector_str>;
}
template <class Archive>
void save(Archive& ar, const doc_2& doc2_value) const
{
ar(CEREAL_NVP(doc2_value));
}
template <class Archive>
void load(Archive& ar, doc_2& doc2_value) const
{
ar(CEREAL_NVP(doc2_value));
}
auto get_pool_name(const vector_one&) const { return "vector_of_ints"; }
auto get_pool_name(const vector_str&) const { return "vector_of_strings"; }
};
The get_pool_types
function returns the types of containers that should be serialized with pools, in this case it’s
both vector
of ints
and strings
. The save
and load
functions control the name of the document node,
in this case it is doc2_value
. And the get_pool_name
overloaded functions supply the name of the pool for each
corresponding immer
container. To create and serialize a value of doc_2
, you can use the following approach:
const auto v1 = vector_one{1, 2, 3};
const auto v2 = v1.push_back(4).push_back(5).push_back(6);
const auto str1 = vector_str{"one", "two"};
const auto str2 =
str1.push_back("three").push_back("four").push_back("five");
const auto value = doc_2{v1, v2, str1, extra_data{str2}};
const auto str =
immer::persist::cereal_save_with_pools(value, doc_2_policy{});
The serialized JSON looks like this:
const auto expected_json = json_t::parse(R"(
{
"doc2_value": {"ints": 0, "ints2": 1, "strings": 0, "extra": {"comments": 1}},
"pools": {
"vector_of_ints": {
"B": 5,
"BL": 1,
"leaves": [[1, [3]], [2, [1, 2]], [4, [5, 6]], [5, [3, 4]]],
"inners": [
[0, {"children": [2], "relaxed": false}],
[3, {"children": [2, 5], "relaxed": false}]
],
"vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
},
"vector_of_strings": {
"B": 5,
"BL": 1,
"leaves": [[1, ["one", "two"]], [3, ["five"]], [4, ["three", "four"]]],
"inners": [
[0, {"children": [], "relaxed": false}],
[2, {"children": [1, 4], "relaxed": false}]
],
"vectors": [{"root": 0, "tail": 1}, {"root": 2, "tail": 3}]
}
}
}
)");
And it can also be loaded from JSON like this:
const auto loaded_value =
immer::persist::cereal_load_with_pools<doc_2>(str, doc_2_policy{});
This example also demonstrates a scenario in which the main document type doc_2
contains another type extra_data
with a
vector
. As you can see in the resulting JSON, nested types are also serialized with pools: "extra": {"comments":
1}
. Only the ID of the comments
vector
is serialized instead of its content.
Suppose, we want to apply certain transforming functions to the immer
containers inside a large document type.
The most straightforward way would be to simply create new containers with the new data, running the transforming
function over each element. However, this approach has some disadvantages:
Let’s consider a simple case using the document from the First example. The desired transformation would be to
multiply each element of the immer::vector<int>
by 10.
First, the document value would be created in the same way:
const auto v1 = vector_one{1, 2, 3};
const auto v2 = v1.push_back(4).push_back(5).push_back(6);
const auto value = document{v1, v2};
The next component we need is the pools of all the containers from the value:
const auto pools = immer::persist::get_output_pools(value);
The get_output_pools
function returns the output pools of all immer
containers that would be serialized using
pools, as controlled by the policy. Here we use the default policy hana_struct_auto_policy
which will use pools for
all immer
containers inside the document type which must be a hana::Struct
.
The other required component is the conversion_map
:
const auto conversion_map = hana::make_map(hana::make_pair(
hana::type_c<vector_one>, [](int val) { return val * 10; }));
This is a hana::map
that describes the desired transformations to be applied. The key of the map is an immer
container and the value is the function to be applied to each element of the corresponding container type. In this case,
it will apply [](int val) { return val * 10; }
to each int
of the vector_one
type, we have two of those in
the document
.
Having these two parts, we can create new pools with the transformations:
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
At this point, we can start converting the immer
containers and create the transformed document value with them,
new_value
:
const auto new_v1 =
immer::persist::convert_container(pools, transformed_pools, v1);
const auto expected_new_v1 = vector_one{10, 20, 30};
REQUIRE(new_v1 == expected_new_v1);
const auto new_v2 =
immer::persist::convert_container(pools, transformed_pools, v2);
const auto expected_new_v2 = vector_one{10, 20, 30, 40, 50, 60};
REQUIRE(new_v2 == expected_new_v2);
const auto new_value = document{new_v1, new_v2};
In order to confirm that the structural sharing has been preserved after applying the transformations, let’s serialize
the new_value
and inspect the JSON:
const auto policy =
immer::persist::hana_struct_auto_member_name_policy(document{});
const auto str =
immer::persist::cereal_save_with_pools(new_value, policy);
const auto expected_json = json_t::parse(R"(
{
"pools": {
"ints": {
"B": 5,
"BL": 1,
"inners": [
[0, {"children": [2], "relaxed": false}],
[3, {"children": [2, 5], "relaxed": false}]
],
"leaves": [[1, [30]], [2, [10, 20]], [4, [50, 60]], [5, [30, 40]]],
"vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
}
},
"value0": {"ints": 0, "ints2": 1}
}
)");
REQUIRE(json_t::parse(str) == expected_json);
And indeed, we can see in the JSON that the node [2, [10, 20]]
is reused in both vectors.
The transforming function can even return a different type. In the following example, vector<int>
is transformed into
vector<std::string>
. The first two steps are the same as in the previous example:
const auto v1 = vector_one{1, 2, 3};
const auto v2 = v1.push_back(4).push_back(5).push_back(6);
const auto value = document{v1, v2};
const auto pools = immer::persist::get_output_pools(value);
Only this time the transforming function will convert an integer into a string:
const auto conversion_map = hana::make_map(hana::make_pair(
hana::type_c<vector_one>,
[](int val) -> std::string { return fmt::format("_{}_", val); }));
Then we convert the two vectors the same way as before:
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
const auto new_v1 =
immer::persist::convert_container(pools, transformed_pools, v1);
const auto expected_new_v1 = vector_str{"_1_", "_2_", "_3_"};
REQUIRE(new_v1 == expected_new_v1);
const auto new_v2 =
immer::persist::convert_container(pools, transformed_pools, v2);
const auto expected_new_v2 =
vector_str{"_1_", "_2_", "_3_", "_4_", "_5_", "_6_"};
REQUIRE(new_v2 == expected_new_v2);
And in order to confirm that the structural sharing has been preserved, we can introduce a new document type with
the two vectors being vector<std::string>
.
namespace {
struct document_str
{
vector_str str;
vector_str str2;
friend bool operator==(const document_str&, const document_str&) = default;
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(str), CEREAL_NVP(str2));
}
};
} // namespace
BOOST_HANA_ADAPT_STRUCT(document_str, str, str2);
And serialize it with pools:
const auto new_value = document_str{new_v1, new_v2};
const auto policy =
immer::persist::hana_struct_auto_member_name_policy(document_str{});
const auto str =
immer::persist::cereal_save_with_pools(new_value, policy);
const auto expected_json = json_t::parse(R"(
{
"pools": {
"str": {
"B": 5,
"BL": 1,
"inners": [
[0, {"children": [2], "relaxed": false}],
[3, {"children": [2, 5], "relaxed": false}]
],
"leaves": [
[1, ["_3_"]],
[2, ["_1_", "_2_"]],
[4, ["_5_", "_6_"]],
[5, ["_3_", "_4_"]]
],
"vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
}
},
"value0": {"str": 0, "str2": 1}
}
)");
REQUIRE(json_t::parse(str) == expected_json);
In the resulting JSON we can confirm that the node [2, ["_1_", "_2_"]]
is reused for both vectors.
As it was shown, converting vectors
is conceptually simple: the transforming function is applied to each element of
each node, producing a new node with the transformed elements. When it comes to the hash-based containers, that is set, map and table, their structure is defined
by the used hash function, so defining the transformation may become a bit more verbose.
In the following example, we’ll start with a simple case of transforming a map. For a map, only the hash of the key
matters and we will not modify the key yet. We will focus on transformations here and not on the structural sharing
within the document, so we will use the immer
container itself as the document. Let’s define the following policy to
indicate that we want to use pools only for our container:
template <class Container>
struct direct_container_policy : immer::persist::value0_serialize_t
{
auto get_pool_types(const auto&) const
{
return boost::hana::tuple_t<Container>;
}
};
By default, immer
uses std::hash
for the hash-based containers. While this hash is sufficient for runtime use, it
can’t be used for persistence, as noted in the C++ reference:
Note
Hash functions are only required to produce the same result for the same input within a single execution of a program
We will use xxHash as the hash for this example. Let’s create a small map like this:
using int_map_t =
immer::map<std::string, int, immer::persist::xx_hash<std::string>>;
const auto value = int_map_t{{"one", 1}, {"two", 2}};
const auto pools = immer::persist::get_output_pools(
value, direct_container_policy<int_map_t>{});
Our goal is to convert the value from int
to std::string
. Let’s create the conversion_map
like this:
namespace hana = boost::hana;
using string_map_t = immer::
map<std::string, std::string, immer::persist::xx_hash<std::string>>;
const auto conversion_map = hana::make_map(hana::make_pair(
hana::type_c<int_map_t>,
hana::overload(
[](const std::pair<std::string, int>& item) {
return std::make_pair(item.first,
fmt::format("_{}_", item.second));
},
[](immer::persist::target_container_type_request) {
return string_map_t{};
})));
A few important details to note:
std::pair<std::string, int>
.immer::persist::target_container_type_request
. This is achieved by using hana::overload
to combine 2 lambdas
into one callable value. When called with that argument, it should return an empty container of the type we’re
transforming to. This explicit approach is necessary because there is no reliable way to automatically determine the
hash algorithm for the new container. Even though in this case the type of the key doesn’t change (and so the hash
remains the same), in other scenarios it might.Once the conversion_map
is defined, the actual conversion is done as before:
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
const auto new_value =
immer::persist::convert_container(pools, transformed_pools, value);
const auto expected_new = string_map_t{{"one", "_1_"}, {"two", "_2_"}};
REQUIRE(new_value == expected_new);
And we can see that the original map’s values have been transformed into strings.
For this example, we’ll transform the type of the ID of the table element while keeping the hash of it the same. This can occur, for instance, if the member that serves as the ID gets wrapped in a wrapper type.
To begin, let’s define an item type for a table:
struct old_item
{
std::string id;
int data;
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(id), CEREAL_NVP(data));
}
};
We can create a table value with some data and get the pools for it like this:
using table_t = immer::table<old_item,
immer::table_key_fn,
immer::persist::xx_hash<std::string>>;
const auto value = table_t{old_item{"one", 1}, old_item{"two", 2}};
const auto pools = immer::persist::get_output_pools(
value, direct_container_policy<table_t>{});
In this example, we want to change the type of the old_item's
ID, which is std::string
, while keeping its hash the same.
Let’s define a wrapper for std::string
and a new_item
type like this:
struct new_id_t
{
std::string id;
friend bool operator==(const new_id_t&, const new_id_t&) = default;
friend std::size_t xx_hash_value(const new_id_t& value)
{
return immer::persist::xx_hash<std::string>{}(value.id);
}
};
struct new_item
{
new_id_t id;
std::string data;
friend bool operator==(const new_item&, const new_item&) = default;
};
We’re also changing the type for data
from int
to std::string
but this change doesn’t affect the structure
of the table. We define the xx_hash_value
function for the new_id_t
type to make it compatible with the
immer::persist::xx_hash<new_id_t>
hash. Then, we can define the target new_table_t
type and the
conversion_map
that describes how to convert old_item
into a new_item
.
using new_table_t = immer::
table<new_item, immer::table_key_fn, immer::persist::xx_hash<new_id_t>>;
const auto conversion_map = hana::make_map(hana::make_pair(
hana::type_c<table_t>,
hana::overload(
[](const old_item& item) {
return new_item{
.id = new_id_t{item.id},
.data = fmt::format("_{}_", item.data),
};
},
[](immer::persist::target_container_type_request) {
return new_table_t{};
})));
Finally, to convert the value
using the defined conversion_map
we prepare the converted pools with
transform_output_pool
and use convert_container
to convert the value
table.
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
const auto new_value =
immer::persist::convert_container(pools, transformed_pools, value);
const auto expected_new =
new_table_t{new_item{{"one"}, "_1_"}, new_item{{"two"}, "_2_"}};
REQUIRE(new_value == expected_new);
We can see that the new_value
table contains the transformed data from the original value
table.
If the key of a map, the ID of a table item or an element of a set changes its hash due to a transformation, the transformed hash-based container can no longer keep its shape and it can’t be efficiently transformed by simply applying transformations to its nodes.
immer::persist
validates every container it creates from a pool. If such a hash modification occurs, a runtime
exception will be thrown because it is not possible to detect this issue during compile-time. Let’s modify the previous
example to also change the data of the ID:
const auto conversion_map = hana::make_map(hana::make_pair(
hana::type_c<table_t>,
hana::overload(
[](const old_item& item) {
return new_item{
// the ID's data is changed and its hash won't be the
// same
.id = new_id_t{item.id + "_key"},
.data = fmt::format("_{}_", item.data),
};
},
[](immer::persist::target_container_type_request) {
return new_table_t{};
})));
Now, if we attempt to convert the original table, a immer::persist::champ::hash_validation_failed_exception
will be thrown:
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
REQUIRE_THROWS_AS(
immer::persist::convert_container(pools, transformed_pools, value),
immer::persist::champ::hash_validation_failed_exception);
Even though such transformation can’t be performed efficiently, on a node level, we can still request these transformations to be applied. This will run for each value of the original container, creating a new independent container that doesn’t use structural sharing:
const auto conversion_map = hana::make_map(hana::make_pair(
hana::type_c<table_t>,
hana::overload(
[](const old_item& item) {
return new_item{
// the ID's data is changed and its hash won't be the
// same
.id = new_id_t{item.id + "_key"},
.data = fmt::format("_{}_", item.data),
};
},
[](immer::persist::target_container_type_request) {
// We know that the hash is changing and requesting to
// transform in a less efficient manner
return immer::persist::incompatible_hash_wrapper<
new_table_t>{};
})));
We can request for such container-level (as opposed to per-node level) transformation to be performed by wrapping the
desired new container type new_table_t
in a immer::persist::incompatible_hash_wrapper
as the result of the
immer::persist::target_container_type_request
call.
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
const auto new_value =
immer::persist::convert_container(pools, transformed_pools, value);
const auto expected_new = new_table_t{new_item{{"one_key"}, "_1_"},
new_item{{"two_key"}, "_2_"}};
REQUIRE(new_value == expected_new);
We can see that the transformation has been applied, the keys have the _key
suffix.
Note
While different transformed containers will not have structural sharing, transforming the same container multiple times will reuse previously transformed data. In other words, transformation will be cached on the container level but not on the nodes level.
const auto new_value_2 =
immer::persist::convert_container(pools, transformed_pools, value);
REQUIRE(new_value_2.impl().root == new_value.impl().root);
Let’s consider a scenario where a transforming function works on an item within an immer
container and also needs to
transform another immer
container. We define the types as follows:
struct nested_t
{
vector_one ints;
friend bool operator==(const nested_t&, const nested_t&) = default;
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(ints));
}
};
struct with_nested_t
{
immer::vector<nested_t> nested;
friend bool operator==(const with_nested_t&,
const with_nested_t&) = default;
template <class Archive>
void serialize(Archive& ar)
{
ar(CEREAL_NVP(nested));
}
};
The important property here is that we have a vector<nested_t>
where nested_t
contains vector<int>
, so we
can say a vector
is nested inside another vector
. We can prepare a value with some structural sharing and then
serialize it:
const auto v1 = vector_one{1, 2, 3};
const auto v2 = v1.push_back(4).push_back(5).push_back(6);
const auto value = with_nested_t{
.nested =
{
nested_t{.ints = v1},
nested_t{.ints = v2},
},
};
const auto policy =
immer::persist::hana_struct_auto_member_name_policy(with_nested_t{});
const auto str = immer::persist::cereal_save_with_pools(value, policy);
The resulting JSON looks like:
const auto expected_json = json_t::parse(R"(
{
"pools": {
"ints": {
"B": 5,
"BL": 1,
"inners": [
[0, {"children": [2], "relaxed": false}],
[3, {"children": [2, 5], "relaxed": false}]
],
"leaves": [[1, [3]], [2, [1, 2]], [4, [5, 6]], [5, [3, 4]]],
"vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
},
"nested": {
"B": 5,
"BL": 3,
"inners": [[0, {"children": [], "relaxed": false}]],
"leaves": [[1, [{"ints": 0}, {"ints": 1}]]],
"vectors": [{"root": 0, "tail": 1}]
}
},
"value0": {"nested": 0}
}
)");
Looking at the JSON we can confirm that the node [2, [1, 2]]
is reused.
Let’s define a conversion_map
like this:
const auto conversion_map = hana::make_map(
hana::make_pair(
hana::type_c<vector_one>,
[](int val) -> std::string { return fmt::format("_{}_", val); }),
hana::make_pair(
hana::type_c<immer::vector<nested_t>>,
[](const nested_t& item, const auto& convert_container) {
return new_nested_t{
.str =
convert_container(hana::type_c<vector_str>, item.ints),
};
}));
The transforming function for vector_one
is simple as it transforms an int
into a std::string
. However, the
function for the vector<nested_t>
is more involved. When we attempt to transform one item of that vector,
nested_t
, we realize that inside that function we have a vector<int>
to deal with. This brings us back to the
problems described in the beginning of the Transformations with pools section. To solve this issue,
immer::persist
provides an optional second argument to the transforming function, a function called
convert_container
. This function can be called with two arguments: the desired container type and the immer
container to convert. This allows us to access the conversion_map
we’re defining. This transformation will be
performed using pools and will preserve structural sharing as expected.
Having defined the conversion_map
, we apply it in the usual way and get the new_value
:
const auto pools = immer::persist::get_output_pools(value, policy);
auto transformed_pools =
immer::persist::transform_output_pool(pools, conversion_map);
const auto new_value = with_new_nested_t{
.nested = immer::persist::convert_container(
pools, transformed_pools, value.nested),
};
We can verify that the new_value
has the expected content:
const auto expected_new = with_new_nested_t{
.nested =
{
new_nested_t{.str = {"_1_", "_2_", "_3_"}},
new_nested_t{.str = {"_1_", "_2_", "_3_", "_4_", "_5_", "_6_"}},
},
};
REQUIRE(new_value == expected_new);
And we can serialize it again to confirm that the structural sharing of the nested vectors has been preserved:
const auto transformed_str = immer::persist::cereal_save_with_pools(
new_value,
immer::persist::hana_struct_auto_member_name_policy(
with_new_nested_t{}));
const auto expected_transformed_json = json_t::parse(R"(
{
"pools": {
"nested": {
"B": 5,
"BL": 3,
"inners": [[0, {"children": [], "relaxed": false}]],
"leaves": [[1, [{"str": 0}, {"str": 1}]]],
"vectors": [{"root": 0, "tail": 1}]
},
"str": {
"B": 5,
"BL": 1,
"inners": [
[0, {"children": [2], "relaxed": false}],
[3, {"children": [2, 5], "relaxed": false}]
],
"leaves": [
[1, ["_3_"]],
[2, ["_1_", "_2_"]],
[4, ["_5_", "_6_"]],
[5, ["_3_", "_4_"]]
],
"vectors": [{"root": 0, "tail": 1}, {"root": 3, "tail": 4}]
}
},
"value0": {"nested": 0}
}
)");
We can see that the [2, ["_1_", "_2_"]]
node is still being reused in the two vectors.
concept immer::persist::Policy
=
requires(Value value, T policy) { policy.get_pool_types(value); }Policy is a type that describes certain aspects of serialization for immer-persist
.
cereal
archive to save and load the user-provided value. Can be used to serealize the value inline (without the value0
node) by taking a dependency on cereal-inline, for example.immer
containers that will be serialized using pools. One pool contains nodes of only one immer
container type.immer::persist::
via_get_pools_names_policy
(const auto &value)¶Create an immer-persist
policy that uses the user-provided get_pools_names
function (located in the same namespace as the value user serializes) to determine:
immer
containers that should be serialized in a poolThe get_pools_names
function is expected to return a boost::hana::map
where key is a container type and value is the name for this container’s pool as a BOOST_HANA_STRING
.
value
: Value that is going to be serialized, only type of the value matters. immer::persist::
hana_struct_auto_member_name_policy
(const auto &value)¶Create an immer-persist
policy that recursively finds all immer
containers in a serialized value and uses member names to name the pools.
The value must be a boost::hana::Struct
.
value
: Value that is going to be serialized, only type of the value matters. immer::persist::
value0_serialize_t
¶This struct provides functions that immer-persist
uses to serialize the user-provided value using cereal
.
In this case, we use cereal
’s default name, value0
. It’s used in all policies provided by immer-persist
.
Other possible way would be to use a third-party library to serialize the value inline (without the value0
node) by taking a dependency on cereal-inline, for example.
Subclassed by immer::persist::hana_struct_auto_member_name_policy_t< T >, immer::persist::via_get_pools_names_policy_t< T >, immer::persist::via_get_pools_types_policy
immer::persist::
demangled_names_t
¶This struct is used in some policies to provide names to each pool by using a demangled name of the immer
container corresponding to the pool.
Subclassed by immer::persist::hana_struct_auto_policy, immer::persist::via_get_pools_types_policy
immer::persist::
via_get_pools_types_policy
¶An immer-persist
policy that uses the user-provided get_pools_types
function to determine the types of immer
containers that should be serialized in a pool.
The get_pools_types
function is expected to return a boost::hana::set
of types of the desired containers.
The names for the pools are determined via demangled_names_t
.
Inherits from immer::persist::demangled_names_t, immer::persist::value0_serialize_t
immer::persist::
hana_struct_auto_policy
¶An immer-persist
policy that recursively finds all immer
containers in a serialized value.
The value must be a boost::hana::Struct
.
The names for the pools are determined via demangled_names_t
.
Inherits from immer::persist::demangled_names_t
immer::persist::
cereal_load_with_pools
(std::istream &is, const Policy &policy = Policy{}, Args&&... args)¶Load a value of the given type T
from the provided stream using pools.
By default, cereal::JSONInputArchive
is used but a different cereal
input archive can be provided.
immer::persist::
cereal_load_with_pools
(const std::string &input, const Policy &policy = Policy{})¶Load a value of the given type T
from the provided string using pools.
By default, cereal::JSONInputArchive
is used but a different cereal
input archive can be provided.
immer::persist::
cereal_save_with_pools
(std::ostream &os, const T &value0, const Policy &policy = Policy{}, Args&&... args)¶Serialize the provided value with pools using the provided policy outputting into the provided stream.
By default, cereal::JSONOutputArchive
is used but a different cereal
output archive can be provided.
immer::persist::
cereal_save_with_pools
(const T &value0, const Policy &policy = Policy{}, Args&&... args)¶Serialize the provided value with pools using the provided policy.
By default, cereal::JSONOutputArchive
is used but a different cereal
output archive can be provided.
immer::persist::
xx_hash
¶xxHash is a good option to be used with immer::persist
as it produces hashes identical across all platforms.
immer::persist::
output_pools_cereal_archive_wrapper
¶A wrapper type that wraps a cereal::OutputArchive
(for example, JSONOutputArchive
), provides access to the Pools
object stored inside, and serializes the pools
object alongside the user document.
Normally, the function cereal_save_with_pools
should be used instead of using this wrapper directly.
immer::persist::
input_pools_cereal_archive_wrapper
¶A wrapper type that wraps a cereal::InputArchive
(for example, JSONInputArchive
) and provides access to the pools
object.
Normally, the function cereal_load_with_pools
should be used instead of using this wrapper directly.
immer::persist::
get_output_pools
(const T &value0, const Policy &policy = Policy{})¶Return just the pools of all the containers of the provided value serialized using the provided policy.
immer::persist::
transform_output_pool
(const detail::output_pools<Storage> &old_pools, const ConversionMap &conversion_map)¶Given output_pools and a map of transformations, produce a new type of input pools with those transformations applied.
conversion_map
is a boost::hana::map
where keys are types of immer
containers and values are the transforming functions.
immer::persist::
convert_container
(const detail::output_pools<SaveStorage> &output_pools, detail::input_pools<LoadStorage> &new_input_pools, const Container &container)¶Given output_pools and new (transformed) input_pools, effectively convert the given container.
immer::persist::
incompatible_hash_wrapper
¶The wrapper is used to enable the incompatible hash mode which is required when the key of a hash-based container transformed in a way that changes its hash.
A value of this type should be returned from a transforming function accepting target_container_type_request
.
immer::persist::
target_container_type_request
¶This type is used as an argument for a transforming function.
The return type of the function is used to specify the desired container type to contain the transformed values.
immer::persist::
pool_exception
¶Base class from which all the exceptions in immer::persist
are derived.
Subclassed by immer::persist::duplicate_name_pool_detected, immer::persist::invalid_children_count, immer::persist::invalid_container_id, immer::persist::invalid_node_id, immer::persist::pool_has_cycles
immer::persist::
pool_has_cycles
¶Thrown when a cycle is detected in the pool of vectors.
Inherits from immer::persist::pool_exception
immer::persist::
invalid_node_id
¶Thrown when a non-existent node is mentioned.
Inherits from immer::persist::pool_exception
immer::persist::
invalid_container_id
¶Thrown when a non-existent container is mentioned.
Inherits from immer::persist::pool_exception
immer::persist::
invalid_children_count
¶Thrown when a node has more children than expected.
Inherits from immer::persist::pool_exception
immer::persist::
duplicate_name_pool_detected
¶Thrown when duplicate pool name is detected.
Inherits from immer::persist::pool_exception