http://hg.omfa.de/prosody-hg/ en-us prosody-hg: util/datamanager.lua history util/datamanager.lua revision history util.datamanager: Fix listing of host stores http://hg.omfa.de/prosody-hg/log/604bed81808e/util/datamanager.lua
lfs returns the basename of the directory entry, and the check neglected to
prefix the data path directory before calling lfs.attribute(), which resulted
in checking Prosody's current working directory for the file instead.]]> Matthew Wild <mwild1@gmail.com> Mon, 25 May 2026 13:23:03 +0100 util.datamanager: Include record length in header http://hg.omfa.de/prosody-hg/log/7ffb2d3ab50b/util/datamanager.lua
In order to distinguish 32bit from 64bit architectures since they look
the same when truncated and little-endian.]]> Kim Alvefur <zash@zash.se> Tue, 09 Dec 2025 21:41:34 +0100 util.datamanager: fix duplicated word in log message http://hg.omfa.de/prosody-hg/log/515ac7e8ae6d/util/datamanager.lua Jonas Schäfer <jonas@wielicki.name> Sat, 16 Nov 2024 09:20:29 +0100 util.datamanager: Fix missing pack format when reading first index entry http://hg.omfa.de/prosody-hg/log/971fd4cb09d4/util/datamanager.lua
Thanks MattJ]]> Kim Alvefur <zash@zash.se> Thu, 30 Nov 2023 18:09:53 +0100 util.datamanager: Load first item into index earlier http://hg.omfa.de/prosody-hg/log/2b8ec22988af/util/datamanager.lua
Should get rid of fseek() call]]> Kim Alvefur <zash@zash.se> Mon, 27 Nov 2023 00:14:56 +0100 util.datamanager: Clean up list index files on purge (i.e. user deletion) http://hg.omfa.de/prosody-hg/log/20b7de7ab66a/util/datamanager.lua Kim Alvefur <zash@zash.se> Thu, 02 Nov 2023 17:35:10 +0100 util.datamanager: Always reset index after list shift http://hg.omfa.de/prosody-hg/log/26327eac56dc/util/datamanager.lua
Shifting the index does not work reliably yet, better to rebuild it from
scratch. Since there is minimal parsing involved in that, it should be
more efficient anyway.]]> Kim Alvefur <zash@zash.se> Sat, 22 Jul 2023 14:02:01 +0200 util.datamanager: Add way to close indexed list store http://hg.omfa.de/prosody-hg/log/9c72f93b7a02/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 21 Jul 2023 18:30:06 +0200 util.datamanager: Close file handle when done using it http://hg.omfa.de/prosody-hg/log/dbd7a6b09ada/util/datamanager.lua
It gets closed eventually but at high load they could potentially
lead to reaching FD limits faster.]]> Kim Alvefur <zash@zash.se> Fri, 21 Jul 2023 18:28:54 +0200 util.datamanager: Disable blockwise removal http://hg.omfa.de/prosody-hg/log/c886ea087b38/util/datamanager.lua
In desperate need of tests]]> Kim Alvefur <zash@zash.se> Fri, 21 Jul 2023 18:28:16 +0200 util.datamanager: Disable block alignment http://hg.omfa.de/prosody-hg/log/6a11d92ae436/util/datamanager.lua
Until we have more test coverage. Somehow the index becomes incorrect
after inserting padding, unclear why.]]> Kim Alvefur <zash@zash.se> Fri, 21 Jul 2023 18:03:22 +0200 util.datamanager: Pad list writes to avoid crossing block boundaries http://hg.omfa.de/prosody-hg/log/affaf6d08d26/util/datamanager.lua
By padding items so that they do not cross block boundaries, it becomes
eaiser to delete whole blocks with fallocate() without cutting items
in half, improving efficiency of such operations.

Since list stores are used for message archives, where the most common
deletion operation would be of the oldest entires, at the top of the
file. With this, all blocks that contain items to be removed could be
deleted without needing to read, delete and write out the whole file.]]> Kim Alvefur <zash@zash.se> Wed, 07 Jun 2023 00:39:30 +0200 util.datamanager: Efficiently remove whole blocks to shift lists http://hg.omfa.de/prosody-hg/log/b57f45165e1e/util/datamanager.lua
Using the new pposix.remove_blocks() it should be very performant to
delete whole sections of a file, given a supporting file system.]]> Kim Alvefur <zash@zash.se> Wed, 12 Jul 2023 11:45:12 +0200 util.datamanager: Add way to efficiently remove first items in a list http://hg.omfa.de/prosody-hg/log/33b114fbb5de/util/datamanager.lua
Copying data without parsing it should be more performant than parsing
it serializing back.]]> Kim Alvefur <zash@zash.se> Wed, 12 Jul 2023 11:42:41 +0200 util.datamanager: Fix indexing first item if not at the very start http://hg.omfa.de/prosody-hg/log/c48ae06e24d6/util/datamanager.lua
If the first item does not start at position 0 then the index function
produces a phantom first entry covering position zero until where the
real first item starts. When using the index, this would make it either
appear as the first item was missing or cause an off-by-one issue with
remaining items.]]> Kim Alvefur <zash@zash.se> Mon, 10 Jul 2023 17:19:05 +0200 util.datamanager: Reduce log level of left over debug messages to debug (thanks Trung) http://hg.omfa.de/prosody-hg/log/87487056bccb/util/datamanager.lua
These were mostly 'warn' to make them stand out from the debug noise]]> Kim Alvefur <zash@zash.se> Wed, 12 Jul 2023 10:24:28 +0200 util.datamanager: Fix missing separator in log line http://hg.omfa.de/prosody-hg/log/48622b89f570/util/datamanager.lua Kim Alvefur <zash@zash.se> Mon, 10 Jul 2023 17:52:52 +0200 util.datamanager: Halve size of list index http://hg.omfa.de/prosody-hg/log/b417a49cc31b/util/datamanager.lua
Instead of storing (start, length) tuples, store the offset to the end
of items and derive length using the previous entry.]]> Kim Alvefur <zash@zash.se> Fri, 14 May 2021 05:49:35 +0200 util.datamanager: Add O(1) list indexing with on-disk index http://hg.omfa.de/prosody-hg/log/638f627e707f/util/datamanager.lua
Index file contains offsets and lengths of each item() which allows
seeking directly to each item and reading it without parsing the entire
file.

Also allows tricks like binary search, assuming items have some defined
order.

We take advantage of the 1-based indexing in tables to store a magic
header in the 0 position, so that table index 1 ends up at file index 1.]]> Kim Alvefur <zash@zash.se> Tue, 11 May 2021 02:09:56 +0200 util: Prefix module imports with prosody namespace http://hg.omfa.de/prosody-hg/log/d10957394a3c/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 17 Mar 2023 16:23:16 +0100 spelling: non-existing mistakes (thanks timeless) http://hg.omfa.de/prosody-hg/log/50fcd3879482/util/datamanager.lua Kim Alvefur <zash@zash.se> Mon, 07 Mar 2022 00:35:29 +0100 Spelling: Fix various spelling mistakes (thanks timeless) http://hg.omfa.de/prosody-hg/log/05c250fa335a/util/datamanager.lua
Words, sometimes I wonder how they even work

Maybe I missed something.]]> Kim Alvefur <zash@zash.se> Mon, 07 Mar 2022 00:13:56 +0100 util.datamanager: Support iterating over any file extension http://hg.omfa.de/prosody-hg/log/87105a9a11df/util/datamanager.lua
The 'typ' argument to all other functions is the actual file extension,
but not here for some reason.

May need this for iterating over the .bin files created by
mod_http_file_share in the future.]]> Kim Alvefur <zash@zash.se> Sun, 07 Feb 2021 19:23:33 +0100 util.datamanager: Fix not passing variables as format string to logger (thanks semgrep) http://hg.omfa.de/prosody-hg/log/5fd1f1b544a0/util/datamanager.lua
Passing error messages as format string could lead to errors, although
not a problem anymore since util.format sanitizes this.]]> Kim Alvefur <zash@zash.se> Sat, 16 Jan 2021 15:49:46 +0100 Merge 0.11->trunk http://hg.omfa.de/prosody-hg/log/dfc8257d8445/util/datamanager.lua Kim Alvefur <zash@zash.se> Mon, 24 Feb 2020 23:33:53 +0100 util.datamanager: Fix iterating over "users" (thanks marc0s) http://hg.omfa.de/prosody-hg/log/69acda92d5a9/util/datamanager.lua

The 'store' path componend used to be unescaped until 756a2a00e7e7 added
escaping to address issues with characters like '/' used in PEP, but
with a special case for '_' which was already in common use in 'store'
path components.

Missed adding this escaping here.]]> Kim Alvefur <zash@zash.se> Mon, 24 Feb 2020 23:16:08 +0100 util.datamanager: Ignore unused 'errno' variable [luacheck] http://hg.omfa.de/prosody-hg/log/810abf34919a/util/datamanager.lua Kim Alvefur <zash@zash.se> Mon, 23 Dec 2019 21:01:55 +0100 Fix various spelling mistakes [codespell] http://hg.omfa.de/prosody-hg/log/02a41315d275/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 03 May 2019 20:54:24 +0200 vairious: Add annotation when an empty environment is set [luacheck] http://hg.omfa.de/prosody-hg/log/4f0f5b49bb03/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 28 Feb 2018 20:06:26 +0100 util.datamanager: Encode the 'store' path component, preserving underscores http://hg.omfa.de/prosody-hg/log/756a2a00e7e7/util/datamanager.lua Kim Alvefur <zash@zash.se> Sun, 08 Oct 2017 18:34:57 +0200 util.datamanager: Use already known index instead of measuring length each iteration http://hg.omfa.de/prosody-hg/log/0a1c0f1107d2/util/datamanager.lua Kim Alvefur <zash@zash.se> Mon, 17 Apr 2017 03:26:21 +0200 util.datamanager: Log where the error came from http://hg.omfa.de/prosody-hg/log/ff3787033abb/util/datamanager.lua Kim Alvefur <zash@zash.se> Tue, 28 Mar 2017 17:31:55 +0200 util.datamanager: Rearrange locals http://hg.omfa.de/prosody-hg/log/72cfbe377326/util/datamanager.lua Kim Alvefur <zash@zash.se> Tue, 28 Mar 2017 17:31:24 +0200 util.datamanager: Use pposix.atomic_append http://hg.omfa.de/prosody-hg/log/f8ba814fe029/util/datamanager.lua Kim Alvefur <zash@zash.se> Tue, 28 Feb 2017 11:33:43 +0100 util.datamanager: Remove check for file non-existence since ENOENT tells us that already http://hg.omfa.de/prosody-hg/log/f4c0fb54e16c/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 22 Mar 2017 12:02:52 +0100 util.datamanager: Split long line [luacheck] http://hg.omfa.de/prosody-hg/log/b99404a340fc/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 22 Mar 2017 12:00:21 +0100 util.datamanager: Expose atomic_store as store_raw http://hg.omfa.de/prosody-hg/log/4017ea6d2599/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 01 Mar 2017 17:37:29 +0100 util.datamanager: Use atomic store function when writing the first entry of a list http://hg.omfa.de/prosody-hg/log/9d014aca766f/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 01 Mar 2017 17:15:56 +0100 util.datamanager: Import value of ENOENT from util.pposix (if available) http://hg.omfa.de/prosody-hg/log/7132abcf669e/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 01 Mar 2017 17:04:48 +0100 util.datamanager: Ignore ENOENT (no such file) when loading data http://hg.omfa.de/prosody-hg/log/209503ee3aaa/util/datamanager.lua Kim Alvefur <zash@zash.se> Wed, 01 Mar 2017 16:44:59 +0100 util.datamanager: Shorter pattern http://hg.omfa.de/prosody-hg/log/6a52415ed68a/util/datamanager.lua Kim Alvefur <zash@zash.se> Sun, 20 Nov 2016 12:06:36 +0100 util.datamanager: Remove extraneous decoding of + to space, which we never never encode to http://hg.omfa.de/prosody-hg/log/601c4d81603b/util/datamanager.lua Kim Alvefur <zash@zash.se> Sun, 20 Nov 2016 12:05:43 +0100 util.datamanager: Use the 'base' argument to tonumber() to indicate hexadecimal http://hg.omfa.de/prosody-hg/log/aba3dd84d9f0/util/datamanager.lua Kim Alvefur <zash@zash.se> Sun, 20 Nov 2016 12:03:24 +0100 util.datamanager: Add annotations to ignore name clashes [luacheck] http://hg.omfa.de/prosody-hg/log/8027eecc750f/util/datamanager.lua Kim Alvefur <zash@zash.se> Sat, 24 Sep 2016 18:31:16 +0200 util.datamanager: Rename variables to avoid name clashes [luacheck] http://hg.omfa.de/prosody-hg/log/177d569307fd/util/datamanager.lua Kim Alvefur <zash@zash.se> Sat, 24 Sep 2016 18:31:00 +0200 util.datamanager: Import tostring and type (fix global access) http://hg.omfa.de/prosody-hg/log/92f721226753/util/datamanager.lua Kim Alvefur <zash@zash.se> Sun, 22 May 2016 14:38:07 +0200 util.datamanager: Explicit handling of each error condition (see #632) http://hg.omfa.de/prosody-hg/log/5bf0ff3882aa/util/datamanager.lua Kim Alvefur <zash@zash.se> Sat, 27 Feb 2016 15:29:56 +0100 Merge 0.9->0.10 http://hg.omfa.de/prosody-hg/log/7a8cffafeff0/util/datamanager.lua Kim Alvefur <zash@zash.se> Thu, 25 Feb 2016 22:37:41 +0100 util.datamanager: Unreference file handle after closing it to prevent trying to close it again (fixes #632) http://hg.omfa.de/prosody-hg/log/67ac4a0b6e50/util/datamanager.lua Kim Alvefur <zash@zash.se> Thu, 25 Feb 2016 22:36:42 +0100 util.datamanager: Add append to public api http://hg.omfa.de/prosody-hg/log/9ab0d5e69c41/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:31:55 +0100 util.datamanager: Return extra location info http://hg.omfa.de/prosody-hg/log/2743759ca1b5/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:31:19 +0100 util.datamanager: Make sure only strings are passed as data to append() http://hg.omfa.de/prosody-hg/log/2b57f77985a3/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:30:39 +0100 util.datamanager: Add some comments about the append function http://hg.omfa.de/prosody-hg/log/0ad66d12113a/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:29:55 +0100 util.datamanager: Handle potential error from :write() call http://hg.omfa.de/prosody-hg/log/86607fe755b6/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:26:04 +0100 util.datamanager: Handle potential issues from fallocate http://hg.omfa.de/prosody-hg/log/0ab228bc21c6/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:24:36 +0100 util.datamanager: No shadowing of variable [luacheck] http://hg.omfa.de/prosody-hg/log/644b1bddc676/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:22:54 +0100 util.datamanager: Skip past second check if first attemtp to open file succeeds http://hg.omfa.de/prosody-hg/log/807b0b2ee545/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:18:24 +0100 util.datamanager: In append() collect status when closing file handle as it may fail (eg the implied flush) http://hg.omfa.de/prosody-hg/log/507301531cf5/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:13:37 +0100 util.datamanager: Overwrite 'data' variable instead of shadownig it [luacheck] http://hg.omfa.de/prosody-hg/log/dc0c6b8dc638/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:11:48 +0100 util.datamanager: Factor out code for appending bytes to a file http://hg.omfa.de/prosody-hg/log/0622f2820d1d/util/datamanager.lua Kim Alvefur <zash@zash.se> Fri, 11 Dec 2015 20:07:22 +0100