annotate mod_storage_s3/mod_storage_s3.lua @ 6257:fa45ae704b79

mod_cloud_notify: Update Readme diff --git a/mod_cloud_notify/README.md b/mod_cloud_notify/README.md --- a/mod_cloud_notify/README.md +++ b/mod_cloud_notify/README.md @@ -1,109 +1,106 @@ ---- -labels: -- 'Stage-Beta' -summary: 'XEP-0357: Cloud push notifications' ---- +# Introduction -Introduction -============ +This module enables support for sending "push notifications" to clients +that need it, typically those running on certain mobile devices. -This module enables support for sending "push notifications" to clients that -need it, typically those running on certain mobile devices. +As well as this module, your client must support push notifications (the +apps that need it generally do, of course) and the app developer's push +gateway must be reachable from your Prosody server (this happens over a +normal XMPP server-to-server 's2s' connection). -As well as this module, your client must support push notifications (the apps -that need it generally do, of course) and the app developer's push gateway -must be reachable from your Prosody server (this happens over a normal XMPP -server-to-server 's2s' connection). - -Details -======= +# Details Some platforms, notably Apple's iOS and many versions of Android, impose -limits that prevent applications from running or accessing the network in the -background. This makes it difficult or impossible for an XMPP application to -remain reliably connected to a server to receive messages. - -In order for messaging and other apps to receive notifications, the OS vendors -run proprietary servers that their OS maintains a permanent connection to in -the background. Then they provide APIs to application developers that allow -sending notifications to specific devices via those servers. +limits that prevent applications from running or accessing the network +in the background. This makes it difficult or impossible for an XMPP +application to remain reliably connected to a server to receive +messages. -When you connect to your server with an app that requires push notifications, -it will use this module to set up a "push registration". When you receive -a message but your device is not connected to the server, this module will -generate a notification and send it to the push gateway operated by your -application's developers). Their gateway will then connect to your device's -OS vendor and ask them to forward the notification to your device. When your -device receives the notification, it will display it or wake up the app so it -can connect to XMPP and receive any pending messages. +In order for messaging and other apps to receive notifications, the OS +vendors run proprietary servers that their OS maintains a permanent +connection to in the background. Then they provide APIs to application +developers that allow sending notifications to specific devices via +those servers. -This protocol is described for developers in [XEP-0357: Push Notifications]. +When you connect to your server with an app that requires push +notifications, it will use this module to set up a "push registration". +When you receive a message but your device is not connected to the +server, this module will generate a notification and send it to the push +gateway operated by your application's developers). Their gateway will +then connect to your device's OS vendor and ask them to forward the +notification to your device. When your device receives the notification, +it will display it or wake up the app so it can connect to XMPP and +receive any pending messages. -For this module to work reliably, you must have [mod_smacks], [mod_mam] and -[mod_carbons] also enabled on your server. +This protocol is described for developers in \[XEP-0357: Push +Notifications\]. + +For this module to work reliably, you must have \[mod_smacks\], +\[mod_mam\] and \[mod_carbons\] also enabled on your server. -Some clients, notably Siskin and Snikket iOS need some additional extensions -that are not currently defined in a standard XEP. To support these clients, -see [mod_cloud_notify_extensions]. +Some clients, notably Siskin and Snikket iOS need some additional +extensions that are not currently defined in a standard XEP. To support +these clients, see \[mod_cloud_notify_extensions\]. -Configuration -============= +# Configuration - Option Default Description - ------------------------------------ ----------------- ------------------------------------------------------------------------------------------------------------------- - `push_notification_important_body` `New Message!` The body text to use when the stanza is important (see above), no message body is sent if this is empty - `push_max_errors` `16` How much persistent push errors are tolerated before notifications for the identifier in question are disabled - `push_max_devices` `5` The number of allowed devices per user (the oldest devices are automatically removed if this threshold is reached) - `push_max_hibernation_timeout` `259200` (72h) Number of seconds to extend the smacks timeout if no push was triggered yet (default: 72 hours) - `push_notification_with_body` (\*) `false` Whether or not to send the real message body to remote pubsub node. Without end-to-end encryption, enabling this may expose your message contents to your client developers and OS vendor. Not recommended. - `push_notification_with_sender` (\*) `false` Whether or not to send the real message sender to remote pubsub node. Enabling this may expose your contacts to your client developers and OS vendor. Not recommended. + Option Default Description + -------------------------------------- ---------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + `push_notification_important_body` `New Message!` The body text to use when the stanza is important (see above), no message body is sent if this is empty + `push_max_errors` `16` How much persistent push errors are tolerated before notifications for the identifier in question are disabled + `push_max_devices` `5` The number of allowed devices per user (the oldest devices are automatically removed if this threshold is reached) + `push_max_hibernation_timeout` `259200` (72h) Number of seconds to extend the smacks timeout if no push was triggered yet (default: 72 hours) + `push_notification_with_body` (\*) `false` Whether or not to send the real message body to remote pubsub node. Without end-to-end encryption, enabling this may expose your message contents to your client developers and OS vendor. Not recommended. + `push_notification_with_sender` (\*) `false` Whether or not to send the real message sender to remote pubsub node. Enabling this may expose your contacts to your client developers and OS vendor. Not recommended. -(\*) There are privacy implications for enabling these options. +(\*) There are privacy implications for enabling these options.[^1] -Internal design notes -===================== +# Internal design notes -App servers are notified about offline messages, messages stored by [mod_mam] -or messages waiting in the smacks queue. -The business rules outlined [here](//mail.jabber.org/pipermail/standards/2016-February/030925.html) are all honored[^2]. +App servers are notified about offline messages, messages stored by +\[mod_mam\] or messages waiting in the smacks queue. The business rules +outlined +[here](//mail.jabber.org/pipermail/standards/2016-February/030925.html) +are all honored[^2]. -To cooperate with [mod_smacks] this module consumes some events: -`smacks-ack-delayed`, `smacks-hibernation-start` and `smacks-hibernation-end`. -These events allow this module to send out notifications for messages received -while the session is hibernated by [mod_smacks] or even when smacks -acknowledgements for messages are delayed by a certain amount of seconds -configurable with the [mod_smacks] setting `smacks_max_ack_delay`. +To cooperate with \[mod_smacks\] this module consumes some events: +`smacks-ack-delayed`, `smacks-hibernation-start` and +`smacks-hibernation-end`. These events allow this module to send out +notifications for messages received while the session is hibernated by +\[mod_smacks\] or even when smacks acknowledgements for messages are +delayed by a certain amount of seconds configurable with the +\[mod_smacks\] setting `smacks_max_ack_delay`. -The `smacks_max_ack_delay` setting allows to send out notifications to clients -which aren't already in smacks hibernation state (because the read timeout or -connection close didn't already happen) but also aren't responding to acknowledgement -request in a timely manner. This setting thus allows conversations to be smoother -under such circumstances. +The `smacks_max_ack_delay` setting allows to send out notifications to +clients which aren't already in smacks hibernation state (because the +read timeout or connection close didn't already happen) but also aren't +responding to acknowledgement request in a timely manner. This setting +thus allows conversations to be smoother under such circumstances. -The new event `cloud-notify-ping` can be used by any module to send out a cloud -notification to either all registered endpoints for the given user or only the endpoints -given in the event data. +The new event `cloud-notify-ping` can be used by any module to send out +a cloud notification to either all registered endpoints for the given +user or only the endpoints given in the event data. -The config setting `push_notification_important_body` can be used to specify an alternative -body text to send to the remote pubsub node if the stanza is encrypted or has a body. -This way the real contents of the message aren't revealed to the push appserver but it -can still see that the push is important. -This is used by Chatsecure on iOS to send out high priority pushes in those cases for example. +The config setting `push_notification_important_body` can be used to +specify an alternative body text to send to the remote pubsub node if +the stanza is encrypted or has a body. This way the real contents of the +message aren't revealed to the push appserver but it can still see that +the push is important. This is used by Chatsecure on iOS to send out +high priority pushes in those cases for example. -Compatibility -============= - -**Note:** This module should be used with Lua 5.2 and higher. Using it with -Lua 5.1 may cause push notifications to not be sent to some clients. +# Compatibility ------- ----------------------------------------------------------------------------- - trunk Works - 0.12 Works - 0.11 Works - 0.10 Works - 0.9 Support dropped, use last supported version [675726ab06d3](//hg.prosody.im/prosody-modules/raw-file/675726ab06d3/mod_cloud_notify/mod_cloud_notify.lua) ------- ----------------------------------------------------------------------------- +**Note:** This module should be used with Lua 5.2 and higher. Using it +with Lua 5.1 may cause push notifications to not be sent to some +clients. + ------- ----------------------------------------------------------------- + trunk Works as of 25-06-13 + 13 Works + 0.12 Works + ------- ----------------------------------------------------------------- -[^1]: The service which is expected to forward notifications to something like Google Cloud Messaging or Apple Notification Service -[^2]: [business_rules.markdown](//hg.prosody.im/prosody-modules/file/tip/mod_cloud_notify/business_rules.markdown) +[^1]: The service which is expected to forward notifications to + something like Google Cloud Messaging or Apple Notification Service + +[^2]: [business_rules.md](//hg.prosody.im/prosody-modules/file/tip/mod_cloud_notify/business_rules.md)
author Menel <menel@snikket.de>
date Fri, 13 Jun 2025 10:36:52 +0200
parents f89367e421d1
children f61564e11d3b
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
1 local http = require "prosody.net.http";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
2 local array = require "prosody.util.array";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
3 local async = require "prosody.util.async";
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
4 local dt = require "prosody.util.datetime";
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
5 local hashes = require "prosody.util.hashes";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
6 local httputil = require "prosody.util.http";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
7 local it = require "prosody.util.iterators";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
8 local jid = require "prosody.util.jid";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
9 local json = require "prosody.util.json";
5697
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
10 local promise = require "prosody.util.promise";
5733
b6518a71ca7e mod_storage_s3: Implement search for set of IDs
Kim Alvefur <zash@zash.se>
parents: 5732
diff changeset
11 local set = require "prosody.util.set";
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
12 local st = require "prosody.util.stanza";
5698
991fb904fa49 mod_storage_s3: Sort imports
Kim Alvefur <zash@zash.se>
parents: 5697
diff changeset
13 local uuid = require "prosody.util.uuid";
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
14 local xml = require "prosody.util.xml";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
15 local url = require "socket.url";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
16
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
17 local new_uuid = uuid.v7 or uuid.generate;
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
18 local hmac_sha256 = hashes.hmac_sha256;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
19 local sha256 = hashes.sha256;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
20
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
21 local driver = {};
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
22
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
23 local bucket = module:get_option_string("s3_bucket", "prosody");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
24 local base_uri = module:get_option_string("s3_base_uri", "http://localhost:9000");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
25 local region = module:get_option_string("s3_region", "us-east-1");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
26
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
27 local access_key = module:get_option_string("s3_access_key");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
28 local secret_key = module:get_option_string("s3_secret_key");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
29
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
30 local aws4_format = "AWS4-HMAC-SHA256 Credential=%s/%s, SignedHeaders=%s, Signature=%s";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
31
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
32 local function aws_auth(event)
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
33 local request, options = event.request, event.options;
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
34 local method = options.method or "GET";
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
35 local query = options.query;
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
36 local payload = options.body;
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
37
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
38 local payload_type = nil;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
39 if st.is_stanza(payload) then
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
40 payload_type = "application/xml";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
41 payload = tostring(payload);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
42 elseif payload ~= nil then
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
43 payload_type = "application/json";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
44 payload = json.encode(payload);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
45 end
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
46 options.body = payload;
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
47
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
48 local payload_hash = sha256(payload or "", true);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
49
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
50 local now = os.time();
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
51 local aws_datetime = os.date("!%Y%m%dT%H%M%SZ", now);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
52 local aws_date = os.date("!%Y%m%d", now);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
53
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
54 local headers = {
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
55 ["Accept"] = "*/*";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
56 ["Authorization"] = nil;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
57 ["Content-Type"] = payload_type;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
58 ["Host"] = request.authority;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
59 ["User-Agent"] = "Prosody XMPP Server";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
60 ["X-Amz-Content-Sha256"] = payload_hash;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
61 ["X-Amz-Date"] = aws_datetime;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
62 };
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
63
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
64 local canonical_uri = url.build({ path = request.path });
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
65 local canonical_query = "";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
66 local canonical_headers = array();
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
67 local signed_headers = array()
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
68
5670
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
69 if query then
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
70 local sorted_query = array();
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
71 for name, value in it.sorted_pairs(query) do
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
72 sorted_query:push({ name = name; value = value });
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
73 end
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
74 sorted_query:sort(function (a,b) return a.name < b.name end)
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
75 canonical_query = httputil.formencode(sorted_query):gsub("%%%x%x", string.upper);
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
76 request.query = canonical_query;
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
77 end
2c9d72ef829e mod_storage_s3: Handle signing of request ?query part
Kim Alvefur <zash@zash.se>
parents: 5669
diff changeset
78
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
79 for header_name, header_value in it.sorted_pairs(headers) do
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
80 header_name = header_name:lower();
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
81 canonical_headers:push(header_name .. ":" .. header_value .. "\n");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
82 signed_headers:push(header_name);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
83 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
84
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
85 canonical_headers = canonical_headers:concat();
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
86 signed_headers = signed_headers:concat(";");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
87
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
88 local scope = aws_date .. "/" .. region .. "/s3/aws4_request";
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
89
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
90 local canonical_request = method .. "\n"
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
91 .. canonical_uri .. "\n"
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
92 .. canonical_query .. "\n"
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
93 .. canonical_headers .. "\n"
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
94 .. signed_headers .. "\n"
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
95 .. payload_hash;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
96
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
97 local signature_payload = "AWS4-HMAC-SHA256" .. "\n" .. aws_datetime .. "\n" .. scope .. "\n" .. sha256(canonical_request, true);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
98
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
99 -- This can be cached?
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
100 local date_key = hmac_sha256("AWS4" .. secret_key, aws_date);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
101 local date_region_key = hmac_sha256(date_key, region);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
102 local date_region_service_key = hmac_sha256(date_region_key, "s3");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
103 local signing_key = hmac_sha256(date_region_service_key, "aws4_request");
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
104
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
105 local signature = hmac_sha256(signing_key, signature_payload, true);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
106
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
107 headers["Authorization"] = string.format(aws4_format, access_key, scope, signed_headers, signature);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
108
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
109 options.headers = headers;
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
110 end
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
111
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
112 function driver:open(store, typ)
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
113 local mt = self[typ or "keyval"]
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
114 if not mt then
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
115 return nil, "unsupported-store";
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
116 end
5950
f89367e421d1 mod_storage_s3: Inherit default HTTP client settings
Kim Alvefur <zash@zash.se>
parents: 5762
diff changeset
117 local httpclient = http.default:new({ connection_pooling = true });
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
118 httpclient.events.add_handler("pre-request", aws_auth);
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
119 return setmetatable({ store = store; bucket = bucket; type = typ; http = httpclient }, mt);
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
120 end
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
121
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
122 local keyval = { };
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
123 driver.keyval = { __index = keyval; __name = module.name .. " keyval store" };
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
124
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
125 local function new_request(self, method, path, query, payload)
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
126 local request = url.parse(base_uri);
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
127 request.path = path;
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
128
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
129 return self.http:request(url.build(request), { method = method; body = payload; query = query });
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
130 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
131
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
132 -- coerce result back into Prosody data type
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
133 local function on_result(response)
5676
799f69a5921a mod_storage_s3: Treat 404 to GET as a signal for empty data
Kim Alvefur <zash@zash.se>
parents: 5675
diff changeset
134 if response.code == 404 and response.request.method == "GET" then
799f69a5921a mod_storage_s3: Treat 404 to GET as a signal for empty data
Kim Alvefur <zash@zash.se>
parents: 5675
diff changeset
135 return nil;
799f69a5921a mod_storage_s3: Treat 404 to GET as a signal for empty data
Kim Alvefur <zash@zash.se>
parents: 5675
diff changeset
136 end
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
137 if response.code >= 400 then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
138 error(response.body);
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
139 end
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
140 local content_type = response.headers["content-type"];
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
141 if content_type == "application/json" then
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
142 return json.decode(response.body);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
143 elseif content_type == "application/xml" then
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
144 return xml.parse(response.body);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
145 elseif content_type == "application/x-www-form-urlencoded" then
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
146 return httputil.formdecode(response.body);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
147 else
5699
83a2fb6df746 mod_storage_s3: Fix logging
Kim Alvefur <zash@zash.se>
parents: 5698
diff changeset
148 module:log("warn", "Unknown response data type %s", content_type);
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
149 return response.body;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
150 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
151 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
152
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
153 function keyval:_path(key)
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
154 return url.build_path({
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
155 is_absolute = true;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
156 bucket;
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
157 jid.escape(module.host);
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
158 jid.escape(key or "@");
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
159 jid.escape(self.store);
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
160 })
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
161 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
162
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
163 function keyval:get(user)
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
164 return async.wait_for(new_request(self, "GET", self:_path(user)):next(on_result));
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
165 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
166
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
167 function keyval:set(user, data)
5671
c8322c64a548 mod_storage_s3: Implement keyvalue deletion
Kim Alvefur <zash@zash.se>
parents: 5670
diff changeset
168
c8322c64a548 mod_storage_s3: Implement keyvalue deletion
Kim Alvefur <zash@zash.se>
parents: 5670
diff changeset
169 if data == nil or (type(data) == "table" and next(data) == nil) then
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
170 return async.wait_for(new_request(self, "DELETE", self:_path(user)));
5671
c8322c64a548 mod_storage_s3: Implement keyvalue deletion
Kim Alvefur <zash@zash.se>
parents: 5670
diff changeset
171 end
c8322c64a548 mod_storage_s3: Implement keyvalue deletion
Kim Alvefur <zash@zash.se>
parents: 5670
diff changeset
172
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
173 return async.wait_for(new_request(self, "PUT", self:_path(user), nil, data));
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
174 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
175
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
176 function keyval:users()
5672
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
177 local bucket_path = url.build_path({ is_absolute = true; bucket; is_directory = true });
5758
32bc648e3892 mod_storage_s3: Fix passing of prefixes, should not be urlencoded
Kim Alvefur <zash@zash.se>
parents: 5737
diff changeset
178 local prefix = jid.escape(module.host) .. "/";
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
179 local list_result, err = async.wait_for(new_request(self, "GET", bucket_path, { prefix = prefix }))
5672
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
180 if err or list_result.code ~= 200 then
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
181 return nil, err;
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
182 end
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
183 local list_bucket_result = xml.parse(list_result.body);
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
184 if list_bucket_result:get_child_text("IsTruncated") == "true" then
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
185 local max_keys = list_bucket_result:get_child_text("MaxKeys");
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
186 module:log("warn", "Paging truncated results not implemented, max %s %s returned", max_keys, self.store);
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
187 end
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
188 local keys = array();
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
189 local store_part = jid.escape(self.store);
5672
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
190 for content in list_bucket_result:childtags("Contents") do
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
191 local key = url.parse_path(content:get_child_text("Key"));
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
192 if key[3] == store_part then
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
193 keys:push(jid.unescape(key[2]));
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
194 end
5672
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
195 end
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
196 return function()
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
197 return keys:pop();
c74a96dc5d58 mod_storage_s3: Implement iteration of keyvalue keys (users usually)
Kim Alvefur <zash@zash.se>
parents: 5671
diff changeset
198 end
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
199 end
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
200
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
201 local archive = {};
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
202 driver.archive = { __index = archive };
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
203
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
204 archive.caps = {
5732
e8938a3166d2 mod_storage_s3: Advertise full id range archive query capability
Kim Alvefur <zash@zash.se>
parents: 5728
diff changeset
205 full_id_range = true; -- both before and after used
5733
b6518a71ca7e mod_storage_s3: Implement search for set of IDs
Kim Alvefur <zash@zash.se>
parents: 5732
diff changeset
206 ids = true;
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
207 };
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
208
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
209 function archive:_path(username, date, when, with, key)
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
210 return url.build_path({
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
211 is_absolute = true;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
212 bucket;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
213 jid.escape(module.host);
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
214 jid.escape(username or "@");
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
215 jid.escape(self.store);
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
216 date or dt.date(when);
5728
80702e33ba71 mod_storage_s3: Fix storing archives for host itself (e.g. mod_audit)
Kim Alvefur <zash@zash.se>
parents: 5700
diff changeset
217 jid.escape(with and jid.prep(with) or "@");
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
218 key;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
219 })
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
220 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
221
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
222
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
223 -- PUT .../with/when/id
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
224 function archive:append(username, key, value, when, with)
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
225 key = key or new_uuid();
5760
59e38aaa3ec1 mod_storage_s3: Remove wrapper and original timestamp from payload (BC)
Kim Alvefur <zash@zash.se>
parents: 5759
diff changeset
226 return async.wait_for(new_request(self, "PUT", self:_path(username, nil, when, with, key), nil, value):next(function(r)
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
227 if r.code == 200 then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
228 return key;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
229 else
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
230 error(r.body);
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
231 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
232 end));
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
233 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
234
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
235 function archive:find(username, query)
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
236 local bucket_path = url.build_path({ is_absolute = true; bucket; is_directory = true });
5758
32bc648e3892 mod_storage_s3: Fix passing of prefixes, should not be urlencoded
Kim Alvefur <zash@zash.se>
parents: 5737
diff changeset
237 local prefix = { jid.escape(module.host); jid.escape(username or "@"); jid.escape(self.store) };
5674
51d0311747fa mod_storage_s3: Handle archive query without parameters
Kim Alvefur <zash@zash.se>
parents: 5673
diff changeset
238 if not query then
51d0311747fa mod_storage_s3: Handle archive query without parameters
Kim Alvefur <zash@zash.se>
parents: 5673
diff changeset
239 query = {};
51d0311747fa mod_storage_s3: Handle archive query without parameters
Kim Alvefur <zash@zash.se>
parents: 5673
diff changeset
240 end
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
241
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
242 if query["start"] and query["end"] and dt.date(query["start"]) == dt.date(query["end"]) then
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
243 table.insert(prefix, dt.date(query["start"]));
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
244 if query["with"] then
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
245 table.insert(prefix, jid.escape(query["with"]));
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
246 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
247 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
248
5758
32bc648e3892 mod_storage_s3: Fix passing of prefixes, should not be urlencoded
Kim Alvefur <zash@zash.se>
parents: 5737
diff changeset
249 prefix = table.concat(prefix, "/").."/";
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
250 local list_result, err = async.wait_for(new_request(self, "GET", bucket_path, {
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
251 prefix = prefix;
5762
ff8b6d0b3bfa mod_storage_s3: Fix mapping archive query limit to ?max-keys=
Kim Alvefur <zash@zash.se>
parents: 5760
diff changeset
252 ["max-keys"] = query["limit"] and tostring(query["limit"]);
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
253 }));
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
254 if err or list_result.code ~= 200 then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
255 return nil, err;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
256 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
257 local list_bucket_result = xml.parse(list_result.body);
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
258 if list_bucket_result:get_child_text("IsTruncated") == "true" then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
259 local max_keys = list_bucket_result:get_child_text("MaxKeys");
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
260 module:log("warn", "Paging truncated results not implemented, max %s %s returned", max_keys, self.store);
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
261 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
262 local keys = array();
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
263 local iterwrap = function(...)
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
264 return ...;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
265 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
266 if query["reverse"] then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
267 query["before"], query["after"] = query["after"], query["before"];
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
268 iterwrap = it.reverse;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
269 end
5733
b6518a71ca7e mod_storage_s3: Implement search for set of IDs
Kim Alvefur <zash@zash.se>
parents: 5732
diff changeset
270 local ids = query["ids"] and set.new(query["ids"]);
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
271 local found = not query["after"];
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
272 for content in iterwrap(list_bucket_result:childtags("Contents")) do
5737
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
273 local date, with, id = table.unpack(url.parse_path(content:get_child_text("Key")), 4);
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
274 local when = dt.parse(content:get_child_text("LastModified"));
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
275 with = jid.unescape(with);
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
276 if found and query["before"] == id then
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
277 break
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
278 end
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
279 if (not query["with"] or query["with"] == with)
5737
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
280 and (not query["start"] or query["start"] <= when)
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
281 and (not query["end"] or query["end"] >= when)
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
282 and (not ids or ids:contains(id))
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
283 and found then
5737
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
284 keys:push({ key = id; date = date; when = when; with = with });
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
285 end
5736
ba731ff5b895 mod_storage_s3: Reorder path components (BC: invalidates any existing data)
Kim Alvefur <zash@zash.se>
parents: 5735
diff changeset
286 if not found and id == query["after"] then
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
287 found = not found
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
288 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
289 end
5737
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
290 keys:sort(function(a, b)
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
291 if a.date ~= b.date then
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
292 return a.date < b.date
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
293 end
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
294 if a.when ~= b.when then
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
295 return a.when < b.when;
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
296 end
5759
c7affea8bb24 mod_storage_s3: Fix sorting items by correct field
Kim Alvefur <zash@zash.se>
parents: 5758
diff changeset
297 return a.key < b.key;
5737
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
298 end);
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
299 if query["reverse"] then
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
300 keys:reverse();
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
301 end
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
302 local i = 0;
5696
66986f5271c3 mod_storage_s3: Skip archive items matching on date but not full datetime
Kim Alvefur <zash@zash.se>
parents: 5695
diff changeset
303 local function get_next()
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
304 i = i + 1;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
305 local item = keys[i];
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
306 if item == nil then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
307 return nil;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
308 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
309 -- luacheck: ignore 431/err
5695
b4632d5f840b mod_storage_s3: Move request signing into a net.http hook
Kim Alvefur <zash@zash.se>
parents: 5676
diff changeset
310 local value, err = async.wait_for(new_request(self, "GET", self:_path(username or "@", item.date, nil, item.with, item.key)):next(on_result));
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
311 if not value then
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
312 module:log("error", "%s", err);
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
313 return nil;
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
314 end
5737
72b0fa7e36dc mod_storage_s3: Sort archive items by LastModified
Kim Alvefur <zash@zash.se>
parents: 5736
diff changeset
315 return item.key, value, item.when, item.with;
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
316 end
5696
66986f5271c3 mod_storage_s3: Skip archive items matching on date but not full datetime
Kim Alvefur <zash@zash.se>
parents: 5695
diff changeset
317 return get_next;
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
318 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
319
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
320 function archive:users()
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
321 return it.unique(keyval.users(self));
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
322 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
323
5697
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
324 local function count(t) local n = 0; for _ in pairs(t) do n = n + 1; end return n; end
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
325
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
326 function archive:delete(username, query)
5697
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
327 local deletions = {};
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
328 for key, _, when, with in self:find(username, query) do
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
329 deletions[key] = new_request(self, "DELETE", self:_path(username or "@", dt.date(when), nil, with, key));
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
330 end
4a0279c5c7ed mod_storage_s3: Implement archive store deletion
Kim Alvefur <zash@zash.se>
parents: 5696
diff changeset
331 return async.wait_for(promise.all(deletions):next(count));
5673
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
332 end
b17ba149b7c5 mod_storage_s3: Implement Archive storage
Kim Alvefur <zash@zash.se>
parents: 5672
diff changeset
333
5669
30f91daa40b4 mod_storage_s3: Beginnings of an experimental S3 storage driver
Kim Alvefur <zash@zash.se>
parents:
diff changeset
334 module:provides("storage", driver);