Stop storing diffs by default in the history

Hi,

As initially suggested by @MichaelHamann, it seems the easiest to fix Loading... would be to just stop storing diff and always store full content for each version (something you can already do with xwiki.store.rcs.nodesPerFull=1 in xwiki.cfg).

I think I heard someone in the back complaining that it’s going to waste tons of space if we do that. The (hopefully enough) answer to that is compression. Currently, we store a full content every 5 versions, so if the compression is good enough to reduce the size to close to 20% it should not eat more space (we might actually win space in some cases if the changes are big enough). The first idea that comes to mind is to compress the data before storing it, but I parsed the documentation of the various databases we officially support and spotted a native compression solution for each one:

• PostgreSQL: it’s compressed by default, but only for content bigger than 2kB (can be changed using the TOAST_TUPLE_THRESHOLD and TOAST_COMPRESSION_THRESHOLD, but the documentation is not super explicit on how to actually set those, might be more obvious for someone more used to PosgreSQL than me) which can happen for a document without any xobject and a small content
• MySQL and MariaDB: it can be enabled when creating the table (if we find a way to tell the hibernate initializer that), or shortly after the creation (on our side) by setting the raw format COMPRESSION (the default is DYNAMIC which also have some storage optimization, but not as much as COMPRESSION)
• Oracle: similarly to MySQL and MariaDB, you can tell Oracle that you would like the table to be COMPRESSED
• HSQLDB: this one is not too critical as it’s not really recommended to use it in production, but it can be enabled in the hibernate.cfg.xml using the property hsqldb.lob_compressed

So as an experiment, I would like to propose, starting with XWiki 17.0.0 (or at least ASAP in 17.x), the following:

• make sure the RCS table has some native compression enabled:
  • doing nothing for PostgreSQL for now (we might decide later to try to force reducing the THRESHOLD)
  • make sure compression is enabled for the table xwikircs in MySQL, MariaDB and Oracle
    • add a migration to convert existing tables
    • make sure the table creation is done with the right flag
• change the default value of xwiki.store.rcs.nodesPerFull to 1, effectively disabling the diff by default (someone can always revert it to 5, or even increase it if not happy with the result in terms of used space)

WDYT ?

Here is my +1.

5 posts - 3 participants

Read full topic