prosody-hg: plugins/mod_storage_internal.lua annotate

annotate plugins/mod_storage_internal.lua @ 14179:25beb66fab38 13.0

util.crypto: Use post-Lua 5.1 buffer API Tha Lua 5.1 buffer API was frustrating to work with. luaL_prepbuffer() only allocates LUAL_BUFFERSIZE - there is no way to request a specific size. LUAL_BUFFERSIZE is set to BUFSIZ by default, which is generally 8192 bytes on Linux, however it may vary between platforms. The variation means this buffer could potentially overflow (small BUFSIZ with large signature). The Lua 5.2+ buffer API enables requesting a specific buffer size, so that's what we do now.

author	Matthew Wild <mwild1@gmail.com>
date	Mon, 25 May 2026 15:11:47 +0100
parents	f77911437539
children

rev	line source
12977 74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	1 local cache = require "prosody.util.cache";
74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	2 local datamanager = require "prosody.core.storagemanager".olddm;
74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	3 local array = require "prosody.util.array";
74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	4 local datetime = require "prosody.util.datetime";
74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	5 local st = require "prosody.util.stanza";
74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	6 local now = require "prosody.util.time".now;
13761 8f516d20d288 mod_storage_internal: Use UUIDv7 for message ids Matthew Wild <mwild1@gmail.com> parents: 13418 diff changeset	7 local uuid_v7 = require "prosody.util.uuid".v7;
12977 74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	8 local jid_join = require "prosody.util.jid".join;
74b9e05af71e plugins: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 11765 diff changeset	9 local set = require "prosody.util.set";
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	10 local it = require "prosody.util.iterators";
4085 7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	11
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	12 local host = module.host;
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	13
13213 50324f66ca2a plugins: Use integer config API with interval specification where sensible Kim Alvefur <zash@zash.se> parents: 13187 diff changeset	14 local archive_item_limit = module:get_option_integer("storage_archive_item_limit", 10000, 0);
13228 616c578c644f mod_storage_internal: Use integer option method for cache size Kim Alvefur <zash@zash.se> parents: 13213 diff changeset	15 local archive_item_count_cache = cache.new(module:get_option_integer("storage_archive_item_limit_cache_size", 1000, 1));
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	16
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	17 local use_shift = module:get_option_boolean("storage_archive_experimental_fast_delete", false);
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	18
5122 b41c33dc7c36 mod_storage_: Don't explicitly set driver name, to ease copying/renaming modules. Waqas Hussain <waqas20@gmail.com>* parents: 5121 diff changeset	19 local driver = {};
4085 7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	20
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5133 diff changeset	21 function driver:open(store, typ)
8018 9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	22 local mt = self[typ or "keyval"]
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	23 if not mt then
6283 7cf6d3a2c855 mod_storage_{none,internal,sql}: Return error for unsupported (everything but keyval) store types Kim Alvefur <zash@zash.se> parents: 5153 diff changeset	24 return nil, "unsupported-store";
7cf6d3a2c855 mod_storage_{none,internal,sql}: Return error for unsupported (everything but keyval) store types Kim Alvefur <zash@zash.se> parents: 5153 diff changeset	25 end
8018 9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	26 return setmetatable({ store = store, type = typ }, mt);
4085 7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	27 end
8018 9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	28
8020 342ce07836de mod_storage_internal: Ignore unused 'self' argument [luacheck] Kim Alvefur <zash@zash.se> parents: 8019 diff changeset	29 function driver:stores(username) -- luacheck: ignore 212/self
8019 925098aad268 mod_storage_internal: Reorder methods Kim Alvefur <zash@zash.se> parents: 8018 diff changeset	30 return datamanager.stores(username, host);
925098aad268 mod_storage_internal: Reorder methods Kim Alvefur <zash@zash.se> parents: 8018 diff changeset	31 end
925098aad268 mod_storage_internal: Reorder methods Kim Alvefur <zash@zash.se> parents: 8018 diff changeset	32
8020 342ce07836de mod_storage_internal: Ignore unused 'self' argument [luacheck] Kim Alvefur <zash@zash.se> parents: 8019 diff changeset	33 function driver:purge(user) -- luacheck: ignore 212/self
8019 925098aad268 mod_storage_internal: Reorder methods Kim Alvefur <zash@zash.se> parents: 8018 diff changeset	34 return datamanager.purge(user, host);
925098aad268 mod_storage_internal: Reorder methods Kim Alvefur <zash@zash.se> parents: 8018 diff changeset	35 end
925098aad268 mod_storage_internal: Reorder methods Kim Alvefur <zash@zash.se> parents: 8018 diff changeset	36
8018 9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	37 local keyval = { };
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	38 driver.keyval = { __index = keyval };
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	39
9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	40 function keyval:get(user)
4085 7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	41 return datamanager.load(user, host, self.store);
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	42 end
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	43
8018 9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	44 function keyval:set(user, data)
4085 7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	45 return datamanager.store(user, host, self.store, data);
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	46 end
7699cef04740 storagemanager, mod_storage_internal: Split out default driver to mod_storage_internal, and greatly simplify storagemanager's error handling and fallback code Matthew Wild <mwild1@gmail.com> parents: diff changeset	47
8018 9545d0a9401f mod_storage_internal: Separate driver from keyval implementation Kim Alvefur <zash@zash.se> parents: 6283 diff changeset	48 function keyval:users()
5153 688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5133 diff changeset	49 return datamanager.users(host, self.store, self.type);
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5133 diff changeset	50 end
688aeac0012a mod_storage_internal, datamanager: Add support for iterating over users with data in a store Kim Alvefur <zash@zash.se> parents: 5133 diff changeset	51
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	52 local archive = {};
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	53 driver.archive = { __index = archive };
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	54
9884 9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set Kim Alvefur <zash@zash.se> parents: 9883 diff changeset	55 archive.caps = {
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set Kim Alvefur <zash@zash.se> parents: 9883 diff changeset	56 total = true;
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set Kim Alvefur <zash@zash.se> parents: 9883 diff changeset	57 quota = archive_item_limit;
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set Kim Alvefur <zash@zash.se> parents: 9883 diff changeset	58 truncate = true;
11275 b8fada57faf0 mod_storage_internal: Add support for full ID range query Kim Alvefur <zash@zash.se> parents: 10926 diff changeset	59 full_id_range = true;
11278 c3907f05bed4 mod_storage_internal: Support query for set of IDs Kim Alvefur <zash@zash.se> parents: 11275 diff changeset	60 ids = true;
9884 9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set Kim Alvefur <zash@zash.se> parents: 9883 diff changeset	61 };
9751c17f5281 mod_storage_internal,_sql: Expose archive capabilities feature set Kim Alvefur <zash@zash.se> parents: 9883 diff changeset	62
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	63 function archive:append(username, key, value, when, with)
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	64 when = when or now();
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	65 if not st.is_stanza(value) then
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	66 return nil, "unsupported-datatype";
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	67 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	68 value = st.preserialize(st.clone(value));
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	69 value.when = when;
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	70 value.with = with;
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	71 value.attr.stamp = datetime.datetime(when);
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	72
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	73 local cache_key = jid_join(username, host, self.store);
64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	74 local item_count = archive_item_count_cache:get(cache_key);
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	75
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	76 if key then
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	77 local items, err = datamanager.list_load(username, host, self.store);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	78 if not items and err then return items, err; end
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	79
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	80 -- Check the quota
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	81 item_count = items and #items or 0;
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	82 archive_item_count_cache:set(cache_key, item_count);
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	83 if item_count >= archive_item_limit then
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	84 module:log("debug", "%s reached or over quota, not adding to store", username);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	85 return nil, "quota-limit";
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	86 end
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	87
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	88 if items then
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	89 -- Filter out any item with the same key as the one being added
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	90 items = array(items);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	91 items:filter(function (item)
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	92 return item.key ~= key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	93 end);
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	94
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	95 value.key = key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	96 items:push(value);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	97 local ok, err = datamanager.list_store(username, host, self.store, items);
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	98 if not ok then return ok, err; end
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	99 archive_item_count_cache:set(cache_key, #items);
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	100 return key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	101 end
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	102 else
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	103 if not item_count then -- Item count not cached?
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	104 -- We need to load the list to get the number of items currently stored
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	105 local items, err = datamanager.list_load(username, host, self.store);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	106 if not items and err then return items, err; end
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	107 item_count = items and #items or 0;
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	108 archive_item_count_cache:set(cache_key, item_count);
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	109 end
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	110 if item_count >= archive_item_limit then
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	111 module:log("debug", "%s reached or over quota, not adding to store", username);
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	112 return nil, "quota-limit";
f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	113 end
13761 8f516d20d288 mod_storage_internal: Use UUIDv7 for message ids Matthew Wild <mwild1@gmail.com> parents: 13418 diff changeset	114 key = uuid_v7();
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	115 end
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	116
9895 6bd65bff03b5 mod_storage_internal: Include store name when reporting quota status Kim Alvefur <zash@zash.se> parents: 9888 diff changeset	117 module:log("debug", "%s has %d items out of %d limit in store %s", username, item_count, archive_item_limit, self.store);
9883 f76bd399267c mod_storage_internal,_sql: Add limit to number of items in an archive store (fixes #733) Matthew Wild <mwild1@gmail.com> parents: 9105 diff changeset	118
8309 5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	119 value.key = key;
5281c479955a mod_storage_internal: Add support for archive key deduplication (like mod_storage_sql) Kim Alvefur <zash@zash.se> parents: 8173 diff changeset	120
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	121 local ok, err = datamanager.list_append(username, host, self.store, value);
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	122 if not ok then return ok, err; end
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	123 archive_item_count_cache:set(cache_key, item_count+1);
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	124 return key;
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	125 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	126
13136 396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	127 local function binary_search(haystack, test, min, max)
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	128 if min == nil then
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	129 min = 1;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	130 end
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	131 if max == nil then
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	132 max = #haystack;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	133 end
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	134
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	135 local floor = math.floor;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	136 while min < max do
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	137 local mid = floor((max + min) / 2);
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	138
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	139 local result = test(haystack[mid]);
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	140 if result < 0 then
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	141 max = mid;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	142 elseif result > 0 then
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	143 min = mid + 1;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	144 else
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	145 return mid, haystack[mid];
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	146 end
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	147 end
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	148
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	149 return min, nil;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	150 end
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	151
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	152 function archive:find(username, query)
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	153 local list, err = datamanager.list_open(username, host, self.store);
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	154 if not list then
8173 3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920) Kim Alvefur <zash@zash.se> parents: 8143 diff changeset	155 if err then
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	156 return list, err;
9997 7c4631d7b6fb mod_storage_internal,memory: Only return total count if requested Kim Alvefur <zash@zash.se> parents: 9903 diff changeset	157 elseif query then
10024 4a0d990253a0 mod_storage_internal: Return appropriate error even with empty archive Kim Alvefur <zash@zash.se> parents: 10018 diff changeset	158 if query.before or query.after then
4a0d990253a0 mod_storage_internal: Return appropriate error even with empty archive Kim Alvefur <zash@zash.se> parents: 10018 diff changeset	159 return nil, "item-not-found";
4a0d990253a0 mod_storage_internal: Return appropriate error even with empty archive Kim Alvefur <zash@zash.se> parents: 10018 diff changeset	160 end
9997 7c4631d7b6fb mod_storage_internal,memory: Only return total count if requested Kim Alvefur <zash@zash.se> parents: 9903 diff changeset	161 if query.total then
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	162 return function()
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	163 end, 0;
9997 7c4631d7b6fb mod_storage_internal,memory: Only return total count if requested Kim Alvefur <zash@zash.se> parents: 9903 diff changeset	164 end
8173 3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920) Kim Alvefur <zash@zash.se> parents: 8143 diff changeset	165 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	166 return function()
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	167 end;
8173 3ff99d49082f mod_storage_internal: Return a noop iterator if archive is empty (fixes #920) Kim Alvefur <zash@zash.se> parents: 8143 diff changeset	168 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	169
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	170 local i = 0;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	171 local iter = function()
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	172 i = i + 1;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	173 return list[i]
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	174 end
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	175
14011 f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	176 local after_found;
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	177 if query then
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	178 if query.reverse then
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	179 i = #list + 1
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	180 iter = function()
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	181 i = i - 1
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	182 return list[i]
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	183 end
13402 6877786d73d7 mod_storage_internal, tests: Fix before/after combined with the 'reverse' flag Matthew Wild <mwild1@gmail.com> parents: 13346 diff changeset	184 query.before, query.after = query.after, query.before;
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	185 end
8089 4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field Kim Alvefur <zash@zash.se> parents: 8023 diff changeset	186 if query.key then
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	187 iter = it.filter(function(item)
8089 4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field Kim Alvefur <zash@zash.se> parents: 8023 diff changeset	188 return item.key == query.key;
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	189 end, iter);
8089 4ba8cb75d925 mod_storage_internal: Support the 'key' archive query field Kim Alvefur <zash@zash.se> parents: 8023 diff changeset	190 end
11278 c3907f05bed4 mod_storage_internal: Support query for set of IDs Kim Alvefur <zash@zash.se> parents: 11275 diff changeset	191 if query.ids then
c3907f05bed4 mod_storage_internal: Support query for set of IDs Kim Alvefur <zash@zash.se> parents: 11275 diff changeset	192 local ids = set.new(query.ids);
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	193 iter = it.filter(function(item)
11278 c3907f05bed4 mod_storage_internal: Support query for set of IDs Kim Alvefur <zash@zash.se> parents: 11275 diff changeset	194 return ids:contains(item.key);
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	195 end, iter);
11278 c3907f05bed4 mod_storage_internal: Support query for set of IDs Kim Alvefur <zash@zash.se> parents: 11275 diff changeset	196 end
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	197 if query.with then
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	198 iter = it.filter(function(item)
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	199 return item.with == query.with;
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	200 end, iter);
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	201 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	202 if query.start then
13136 396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	203 if not query.reverse then
13418 2374c7665d0b mod_storage_internal: Fix off-by-one when searching archive for Kim Alvefur <zash@zash.se> parents: 13402 diff changeset	204 local wi = binary_search(list, function(item)
13136 396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	205 local when = item.when or datetime.parse(item.attr.stamp);
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	206 return query.start - when;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	207 end);
13418 2374c7665d0b mod_storage_internal: Fix off-by-one when searching archive for Kim Alvefur <zash@zash.se> parents: 13402 diff changeset	208 i = wi - 1;
13136 396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	209 end
13847 cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	210 iter = it.filter(function(item)
cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	211 local when = item.when or datetime.parse(item.attr.stamp);
cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	212 return when >= query.start;
cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	213 end, iter);
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	214 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	215 if query["end"] then
13136 396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	216 if query.reverse then
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	217 local wi = binary_search(list, function(item)
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	218 local when = item.when or datetime.parse(item.attr.stamp);
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	219 return query["end"] - when;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	220 end);
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	221 if wi then
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	222 i = wi + 1;
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	223 end
396db0e7084f mod_storage_internal: Use a binary search for time based ranges Kim Alvefur <zash@zash.se> parents: 13135 diff changeset	224 end
13847 cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	225 iter = it.filter(function(item)
cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	226 local when = item.when or datetime.parse(item.attr.stamp);
cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	227 return when <= query["end"];
cbd234461c41 mod_storage_internal: Fix queries with only start returning extra items Kim Alvefur <zash@zash.se> parents: 13761 diff changeset	228 end, iter);
9997 7c4631d7b6fb mod_storage_internal,memory: Only return total count if requested Kim Alvefur <zash@zash.se> parents: 9903 diff changeset	229 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	230 if query.after then
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	231 local found = false;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	232 iter = it.filter(function(item)
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	233 local found_after = found;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	234 if item.key == query.after then
14011 f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	235 after_found = true;
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	236 found = true
10018 7408b9473729 mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325) Kim Alvefur <zash@zash.se> parents: 10001 diff changeset	237 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	238 return found_after;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	239 end, iter);
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	240 end
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	241 if query.before then
14011 f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	242 local before_key, before_found = query.before;
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	243 for idx = 1, #list do
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	244 if list[idx].key == before_key then
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	245 before_found = true;
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	246 break;
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	247 end
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	248 end
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	249 if not before_found then
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	250 return nil, "item-not-found";
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	251 end
10018 7408b9473729 mod_storage_internal: Return error if 'before' or 'after' are not found (partial fix for #1325) Kim Alvefur <zash@zash.se> parents: 10001 diff changeset	252 local found = false;
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	253 iter = it.filter(function(item)
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	254 if item.key == query.before then
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	255 found = true
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	256 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	257 return not found;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	258 end, iter);
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	259 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	260 if query.limit then
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	261 iter = it.head(query.limit, iter);
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	262 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	263 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	264
14011 f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	265 local first_item = iter();
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	266 if first_item == nil and query and query.after and not after_found then
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	267 return nil, "item-not-found";
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	268 end
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	269
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	270 return function()
14011 f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	271 local item;
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	272 if first_item then
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	273 item, first_item = first_item, nil;
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	274 else
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	275 item = iter();
f77911437539 mod_storage_internal: Return item-not-found for unknown before/after ids Matthew Wild <mwild1@gmail.com> parents: 13847 diff changeset	276 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	277 if item == nil then
13343 9a9455de295d mod_storage_internal: Only close lazy-loading list store abstractions Kim Alvefur <zash@zash.se> parents: 13340 diff changeset	278 if list.close then
9a9455de295d mod_storage_internal: Only close lazy-loading list store abstractions Kim Alvefur <zash@zash.se> parents: 13340 diff changeset	279 list:close();
9a9455de295d mod_storage_internal: Only close lazy-loading list store abstractions Kim Alvefur <zash@zash.se> parents: 13340 diff changeset	280 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	281 return
10926 c55bd98a54f8 mod_storage_internal, mod_storage_memory: Add support for query.before Matthew Wild <mwild1@gmail.com> parents: 10844 diff changeset	282 end
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	283 local key = item.key;
3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	284 local when = item.when or item.attr and datetime.parse(item.attr.stamp);
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	285 local with = item.with;
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	286 item.key, item.when, item.with = nil, nil, nil;
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	287 item.attr.stamp = nil;
11765 1cac469b18d0 mod_storage_internal: Stop storing XEP-0091 timestamp Kim Alvefur <zash@zash.se> parents: 11278 diff changeset	288 -- COMPAT Stored data may still contain legacy XEP-0091 timestamp
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	289 item.attr.stamp_legacy = nil;
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	290 item = st.deserialize(item);
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	291 return key, item, when, with;
13135 3fd24e1945b0 mod_storage_internal: Lazy-load archive items while iterating Kim Alvefur <zash@zash.se> parents: 12977 diff changeset	292 end
8021 83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	293 end
83f18982bcfd mod_storage_internal: Add basic archive store implementation Kim Alvefur <zash@zash.se> parents: 8020 diff changeset	294
10837 f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	295 function archive:get(username, wanted_key)
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	296 local iter, err = self:find(username, { key = wanted_key })
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	297 if not iter then return iter, err; end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	298 for key, stanza, when, with in iter do
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	299 if key == wanted_key then
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	300 return stanza, when, with;
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	301 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	302 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	303 return nil, "item-not-found";
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	304 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	305
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	306 function archive:set(username, key, new_value, new_when, new_with)
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	307 local items, err = datamanager.list_load(username, host, self.store);
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	308 if not items then
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	309 if err then
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	310 return items, err;
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	311 else
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	312 return nil, "item-not-found";
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	313 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	314 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	315
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	316 for i = 1, #items do
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	317 local old_item = items[i];
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	318 if old_item.key == key then
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	319 local item = st.preserialize(st.clone(new_value));
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	320
10842 5a6ba2f38e2b mod_storage_internal: Fix keeping old timestamp in archive map API Kim Alvefur <zash@zash.se> parents: 10837 diff changeset	321 local when = new_when or old_item.when or datetime.parse(old_item.attr.stamp);
10837 f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	322 item.key = key;
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	323 item.when = when;
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	324 item.with = new_with or old_item.with;
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	325 item.attr.stamp = datetime.datetime(when);
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	326 items[i] = item;
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	327 return datamanager.list_store(username, host, self.store, items);
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	328 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	329 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	330
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	331 return nil, "item-not-found";
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	332 end
f23363380599 mod_storage_internal: Implement key-value API Kim Alvefur <zash@zash.se> parents: 10223 diff changeset	333
8023 5a9d491cc714 mod_storage_internal: Add the dates method Kim Alvefur <zash@zash.se> parents: 8022 diff changeset	334 function archive:dates(username)
5a9d491cc714 mod_storage_internal: Add the dates method Kim Alvefur <zash@zash.se> parents: 8022 diff changeset	335 local items, err = datamanager.list_load(username, host, self.store);
5a9d491cc714 mod_storage_internal: Add the dates method Kim Alvefur <zash@zash.se> parents: 8022 diff changeset	336 if not items then return items, err; end
5a9d491cc714 mod_storage_internal: Add the dates method Kim Alvefur <zash@zash.se> parents: 8022 diff changeset	337 return array(items):pluck("when"):map(datetime.date):unique();
5a9d491cc714 mod_storage_internal: Add the dates method Kim Alvefur <zash@zash.se> parents: 8022 diff changeset	338 end
5a9d491cc714 mod_storage_internal: Add the dates method Kim Alvefur <zash@zash.se> parents: 8022 diff changeset	339
9903 2c5546cc5c70 mod_storage_internal: Implement a summary API returning message counts per contact Kim Alvefur <zash@zash.se> parents: 9895 diff changeset	340 function archive:summary(username, query)
2c5546cc5c70 mod_storage_internal: Implement a summary API returning message counts per contact Kim Alvefur <zash@zash.se> parents: 9895 diff changeset	341 local iter, err = self:find(username, query)
2c5546cc5c70 mod_storage_internal: Implement a summary API returning message counts per contact Kim Alvefur <zash@zash.se> parents: 9895 diff changeset	342 if not iter then return iter, err; end
10220 1e2b444acb72 mod_storage_: Tweak :summary API to allow future expansion with more fields Kim Alvefur <zash@zash.se>* parents: 10024 diff changeset	343 local counts = {};
10222 51f145094648 mod_storage_: Also include timestmap of first message in :summary API Kim Alvefur <zash@zash.se>* parents: 10221 diff changeset	344 local earliest = {};
10221 068692cb9e78 mod_storage_: Include timestamp of latest message in :summary API Kim Alvefur <zash@zash.se>* parents: 10220 diff changeset	345 local latest = {};
10223 d185c4961ee0 mod_storage_internal: Include last text message Kim Alvefur <zash@zash.se> parents: 10222 diff changeset	346 local body = {};
d185c4961ee0 mod_storage_internal: Include last text message Kim Alvefur <zash@zash.se> parents: 10222 diff changeset	347 for _, stanza, when, with in iter do
10220 1e2b444acb72 mod_storage_: Tweak :summary API to allow future expansion with more fields Kim Alvefur <zash@zash.se>* parents: 10024 diff changeset	348 counts[with] = (counts[with] or 0) + 1;
10222 51f145094648 mod_storage_: Also include timestmap of first message in :summary API Kim Alvefur <zash@zash.se>* parents: 10221 diff changeset	349 if earliest[with] == nil then
51f145094648 mod_storage_: Also include timestmap of first message in :summary API Kim Alvefur <zash@zash.se>* parents: 10221 diff changeset	350 earliest[with] = when;
51f145094648 mod_storage_: Also include timestmap of first message in :summary API Kim Alvefur <zash@zash.se>* parents: 10221 diff changeset	351 end
10221 068692cb9e78 mod_storage_: Include timestamp of latest message in :summary API Kim Alvefur <zash@zash.se>* parents: 10220 diff changeset	352 latest[with] = when;
10223 d185c4961ee0 mod_storage_internal: Include last text message Kim Alvefur <zash@zash.se> parents: 10222 diff changeset	353 body[with] = stanza:get_child_text("body") or body[with];
9903 2c5546cc5c70 mod_storage_internal: Implement a summary API returning message counts per contact Kim Alvefur <zash@zash.se> parents: 9895 diff changeset	354 end
10220 1e2b444acb72 mod_storage_: Tweak :summary API to allow future expansion with more fields Kim Alvefur <zash@zash.se>* parents: 10024 diff changeset	355 return {
1e2b444acb72 mod_storage_: Tweak :summary API to allow future expansion with more fields Kim Alvefur <zash@zash.se>* parents: 10024 diff changeset	356 counts = counts;
10222 51f145094648 mod_storage_: Also include timestmap of first message in :summary API Kim Alvefur <zash@zash.se>* parents: 10221 diff changeset	357 earliest = earliest;
10221 068692cb9e78 mod_storage_: Include timestamp of latest message in :summary API Kim Alvefur <zash@zash.se>* parents: 10220 diff changeset	358 latest = latest;
10223 d185c4961ee0 mod_storage_internal: Include last text message Kim Alvefur <zash@zash.se> parents: 10222 diff changeset	359 body = body;
10220 1e2b444acb72 mod_storage_: Tweak :summary API to allow future expansion with more fields Kim Alvefur <zash@zash.se>* parents: 10024 diff changeset	360 };
9903 2c5546cc5c70 mod_storage_internal: Implement a summary API returning message counts per contact Kim Alvefur <zash@zash.se> parents: 9895 diff changeset	361 end
2c5546cc5c70 mod_storage_internal: Implement a summary API returning message counts per contact Kim Alvefur <zash@zash.se> parents: 9895 diff changeset	362
10001 55f5588d71c6 mod_storage_internal: Add support for iterating over users in archive stores Kim Alvefur <zash@zash.se> parents: 9997 diff changeset	363 function archive:users()
55f5588d71c6 mod_storage_internal: Add support for iterating over users in archive stores Kim Alvefur <zash@zash.se> parents: 9997 diff changeset	364 return datamanager.users(host, self.store, "list");
55f5588d71c6 mod_storage_internal: Add support for iterating over users in archive stores Kim Alvefur <zash@zash.se> parents: 9997 diff changeset	365 end
55f5588d71c6 mod_storage_internal: Add support for iterating over users in archive stores Kim Alvefur <zash@zash.se> parents: 9997 diff changeset	366
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	367 function archive:trim(username, to_when)
13346 53c347043cb5 mod_storage_internal: Clear archive item count cache after experimental trim Kim Alvefur <zash@zash.se> parents: 13344 diff changeset	368 local cache_key = jid_join(username, host, self.store);
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	369 local list, err = datamanager.list_open(username, host, self.store);
13263 e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	370 if not list then
e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	371 if err == nil then
e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	372 module:log("debug", "store already empty, can't trim");
e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	373 return 0;
e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	374 end
e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	375 return list, err;
e77994e88940 mod_storage_internal: Don't report error when attempting to trim empty archive Kim Alvefur <zash@zash.se> parents: 13262 diff changeset	376 end
13262 9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	377
9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	378 -- shortcut: check if the last item should be trimmed, if so, drop the whole archive
9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	379 local last = list[#list].when or datetime.parse(list[#list].attr.stamp);
9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	380 if last <= to_when then
13344 958c759d3897 mod_storage_internal: Close lazy-loading list abstraction after trim Kim Alvefur <zash@zash.se> parents: 13343 diff changeset	381 if list.close then
958c759d3897 mod_storage_internal: Close lazy-loading list abstraction after trim Kim Alvefur <zash@zash.se> parents: 13343 diff changeset	382 list:close()
958c759d3897 mod_storage_internal: Close lazy-loading list abstraction after trim Kim Alvefur <zash@zash.se> parents: 13343 diff changeset	383 end
13262 9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	384 return datamanager.list_store(username, host, self.store, nil);
9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	385 end
9a86e7cbdd79 mod_storage_internal: Fix fast trimming of archive with exactly one item Kim Alvefur <zash@zash.se> parents: 13228 diff changeset	386
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	387 -- luacheck: ignore 211/exact
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	388 local i, exact = binary_search(list, function(item)
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	389 local when = item.when or datetime.parse(item.attr.stamp);
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	390 return to_when - when;
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	391 end);
13344 958c759d3897 mod_storage_internal: Close lazy-loading list abstraction after trim Kim Alvefur <zash@zash.se> parents: 13343 diff changeset	392 if list.close then
958c759d3897 mod_storage_internal: Close lazy-loading list abstraction after trim Kim Alvefur <zash@zash.se> parents: 13343 diff changeset	393 list:close()
958c759d3897 mod_storage_internal: Close lazy-loading list abstraction after trim Kim Alvefur <zash@zash.se> parents: 13343 diff changeset	394 end
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	395 -- TODO if exact then ... off by one?
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	396 if i == 1 then return 0; end
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	397 local ok, err = datamanager.list_shift(username, host, self.store, i);
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	398 if not ok then return ok, err; end
13346 53c347043cb5 mod_storage_internal: Clear archive item count cache after experimental trim Kim Alvefur <zash@zash.se> parents: 13344 diff changeset	399 archive_item_count_cache:set(cache_key, nil); -- TODO calculate how many items are left
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	400 return i-1;
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	401 end
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	402
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	403 function archive:delete(username, query)
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	404 local cache_key = jid_join(username, host, self.store);
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	405 if not query or next(query) == nil then
13346 53c347043cb5 mod_storage_internal: Clear archive item count cache after experimental trim Kim Alvefur <zash@zash.se> parents: 13344 diff changeset	406 archive_item_count_cache:set(cache_key, nil); -- nil because we don't check if the following succeeds
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	407 return datamanager.list_store(username, host, self.store, nil);
05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	408 end
13187 fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	409
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	410 if use_shift and next(query) == "end" and next(query, "end") == nil then
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	411 return self:trim(username, query["end"]);
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	412 end
fe1229919070 mod_storage_internal: Implement efficient deletion of oldest archive items Kim Alvefur <zash@zash.se> parents: 13136 diff changeset	413
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	414 local items, err = datamanager.list_load(username, host, self.store);
8143 c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	415 if not items then
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	416 if err then
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	417 return items, err;
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	418 end
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	419 archive_item_count_cache:set(cache_key, 0);
8143 c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	420 -- Store is empty
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	421 return 0;
c4c159953c72 mod_storage_internal: Handle case of empty item store when deleting (fixes #910) Kim Alvefur <zash@zash.se> parents: 8142 diff changeset	422 end
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	423 items = array(items);
8142 13e075549cb3 mod_storage_internal: Correctly calculate number of deleted items (fixes #912) Kim Alvefur <zash@zash.se> parents: 8089 diff changeset	424 local count_before = #items;
8310 1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	425 if query then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	426 if query.key then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	427 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	428 return item.key ~= query.key;
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	429 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	430 end
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	431 if query.with then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	432 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	433 return item.with ~= query.with;
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	434 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	435 end
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	436 if query.start then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	437 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	438 return item.when < query.start;
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	439 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	440 end
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	441 if query["end"] then
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	442 items:filter(function (item)
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	443 return item.when > query["end"];
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	444 end);
1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	445 end
9105 e735c9865f42 mod_storage_internal: Only apply truncate if there are more items than requested Kim Alvefur <zash@zash.se> parents: 8402 diff changeset	446 if query.truncate and #items > query.truncate then
8393 b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	447 if query.reverse then
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	448 -- Before: { 1, 2, 3, 4, 5, }
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	449 -- After: { 1, 2, 3 }
8394 4892c22403d5 mod_storage_internal: Optimize truncation Kim Alvefur <zash@zash.se> parents: 8393 diff changeset	450 for i = #items, query.truncate + 1, -1 do
4892c22403d5 mod_storage_internal: Optimize truncation Kim Alvefur <zash@zash.se> parents: 8393 diff changeset	451 items[i] = nil;
8393 b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	452 end
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	453 else
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	454 -- Before: { 1, 2, 3, 4, 5, }
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	455 -- After: { 3, 4, 5 }
8394 4892c22403d5 mod_storage_internal: Optimize truncation Kim Alvefur <zash@zash.se> parents: 8393 diff changeset	456 local offset = #items - query.truncate;
4892c22403d5 mod_storage_internal: Optimize truncation Kim Alvefur <zash@zash.se> parents: 8393 diff changeset	457 for i = 1, #items do
4892c22403d5 mod_storage_internal: Optimize truncation Kim Alvefur <zash@zash.se> parents: 8393 diff changeset	458 items[i] = items[i+offset];
8393 b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	459 end
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	460 end
b6a7b83f8d87 mod_storage_internal: Allow truncating deletion at the beginning or end of an archive store Kim Alvefur <zash@zash.se> parents: 8310 diff changeset	461 end
8310 1759491b53db mod_storage_internal: Add more extensive query support to archive:delete method Kim Alvefur <zash@zash.se> parents: 8309 diff changeset	462 end
8142 13e075549cb3 mod_storage_internal: Correctly calculate number of deleted items (fixes #912) Kim Alvefur <zash@zash.se> parents: 8089 diff changeset	463 local count = count_before - #items;
8402 469afa02947b mod_storage_internal: Skip write if no items matched a deletion query Kim Alvefur <zash@zash.se> parents: 8394 diff changeset	464 if count == 0 then
469afa02947b mod_storage_internal: Skip write if no items matched a deletion query Kim Alvefur <zash@zash.se> parents: 8394 diff changeset	465 return 0; -- No changes, skip write
469afa02947b mod_storage_internal: Skip write if no items matched a deletion query Kim Alvefur <zash@zash.se> parents: 8394 diff changeset	466 end
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	467 local ok, err = datamanager.list_store(username, host, self.store, items);
05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	468 if not ok then return ok, err; end
9885 64e16d1e91f6 mod_storage_internal,_sql: Key item count cache on both username and store Kim Alvefur <zash@zash.se> parents: 9884 diff changeset	469 archive_item_count_cache:set(cache_key, #items);
8022 05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	470 return count;
05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	471 end
05e201468f27 mod_storage_internal: Add support for removing archived items Kim Alvefur <zash@zash.se> parents: 8021 diff changeset	472
5121 b5a5643f8572 core.storagemanager, mod_storage_: "data-driver" -> "storage-provider", to allow using module:provides(). Waqas Hussain <waqas20@gmail.com>* parents: 5039 diff changeset	473 module:provides("storage", driver);

Mercurial > prosody-hg

annotate plugins/mod_storage_internal.lua @ 14179:25beb66fab38 13.0