prosody-hg: net/http.lua annotate

annotate net/http.lua @ 13319:6d6291dfe735

net.http: Add simple connection pooling This should speed up repeated requests to the same site by keeping their connections around and sending more requests on them. Sending multiple requests at the same time is not supported, instead a request started while another to the same authority is in progress would open a new one and the first one to complete would go back in the pool. This could be investigated in the future. Some http servers limit the number of requests per connection and this is not tested and could cause one request to fail, but hopefully it will close the connection and prevent it from being reused.

author	Kim Alvefur <zash@zash.se>
date	Sat, 11 Nov 2023 23:08:34 +0100
parents	ba409c67353b
children	23f95714c386

rev	line source
1522 569d58d21612 Add copyright header to those files missing one Matthew Wild <mwild1@gmail.com> parents: 1331 diff changeset	1 -- Prosody IM
2923 b7049746bd29 Update copyright headers for 2010 Matthew Wild <mwild1@gmail.com> parents: 2810 diff changeset	2 -- Copyright (C) 2008-2010 Matthew Wild
b7049746bd29 Update copyright headers for 2010 Matthew Wild <mwild1@gmail.com> parents: 2810 diff changeset	3 -- Copyright (C) 2008-2010 Waqas Hussain
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	4 --
1522 569d58d21612 Add copyright header to those files missing one Matthew Wild <mwild1@gmail.com> parents: 1331 diff changeset	5 -- This project is MIT/X11 licensed. Please see the
569d58d21612 Add copyright header to those files missing one Matthew Wild <mwild1@gmail.com> parents: 1331 diff changeset	6 -- COPYING file in the source package for more information.
569d58d21612 Add copyright header to those files missing one Matthew Wild <mwild1@gmail.com> parents: 1331 diff changeset	7 --
569d58d21612 Add copyright header to those files missing one Matthew Wild <mwild1@gmail.com> parents: 1331 diff changeset	8
12974 ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	9 local b64 = require "prosody.util.encodings".base64.encode;
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	10 local url = require "socket.url"
12974 ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	11 local httpstream_new = require "prosody.net.http.parser".new;
ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	12 local util_http = require "prosody.util.http";
ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	13 local events = require "prosody.util.events";
ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	14 local verify_identity = require"prosody.util.x509".verify_identity;
ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	15 local promise = require "prosody.util.promise";
ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	16 local http_errors = require "prosody.net.http.errors";
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	17
12974 ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	18 local basic_resolver = require "prosody.net.resolvers.basic";
ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	19 local connect = require "prosody.net.connect".connect;
5448 cbe9fa2d3787 net.http: Throw error when connecting to a http:// URL without LuaSec available Matthew Wild <mwild1@gmail.com> parents: 5354 diff changeset	20
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	21 local ssl_available = pcall(require, "ssl");
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	22
69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	23 local t_insert, t_concat = table.insert, table.concat;
5505 0b6a99e6c1b1 mod_c2s, mod_s2s, net.http, net.http.server: Improve tracebacks (omit traceback function), to make it clearer where an error occured Matthew Wild <mwild1@gmail.com> parents: 5488 diff changeset	24 local pairs = pairs;
9562 acf74ad0b795 Many things: switch from hacky multi-arg xpcall implementations to a standard util.xpcall Matthew Wild <mwild1@gmail.com> parents: 8731 diff changeset	25 local tonumber, tostring, traceback =
acf74ad0b795 Many things: switch from hacky multi-arg xpcall implementations to a standard util.xpcall Matthew Wild <mwild1@gmail.com> parents: 8731 diff changeset	26 tonumber, tostring, debug.traceback;
11220 9b25eecde9e6 net.http: track time of request for debug/stats purposes Matthew Wild <mwild1@gmail.com> parents: 11185 diff changeset	27 local os_time = os.time;
12974 ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	28 local xpcall = require "prosody.util.xpcall".xpcall;
7792 0bc6c3704973 net.http: Remove unused imports [luacheck] Kim Alvefur <zash@zash.se> parents: 7520 diff changeset	29 local error = error
678 1859edec2237 Protected call for HTTP request callbacks, to catch errors Matthew Wild <mwild1@gmail.com> parents: 677 diff changeset	30
12974 ba409c67353b net: Prefix module imports with prosody namespace Kim Alvefur <zash@zash.se> parents: 12881 diff changeset	31 local log = require "prosody.util.logger".init("http");
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	32
6780 647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	33 local _ENV = nil;
8555 4f0f5b49bb03 vairious: Add annotation when an empty environment is set [luacheck] Kim Alvefur <zash@zash.se> parents: 8551 diff changeset	34 -- luacheck: std none
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	35
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	36 local requests = {}; -- Open requests
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	37
7463 3b6e7ce9431f net.http: Add request.id to every request object (can be overridden by providing ex.id) Matthew Wild <mwild1@gmail.com> parents: 6823 diff changeset	38 local function make_id(req) return (tostring(req):match("%x+$")); end
3b6e7ce9431f net.http: Add request.id to every request object (can be overridden by providing ex.id) Matthew Wild <mwild1@gmail.com> parents: 6823 diff changeset	39
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	40 local listener = { default_port = 80, default_mode = "*a" };
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	41
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	42 -- Request-related helper functions
8689 c7122fbe1600 net.http: Fix for Lua 5.2: return error from error handler (xpcall changed) Matthew Wild <mwild1@gmail.com> parents: 8555 diff changeset	43 local function handleerr(err) log("error", "Traceback[http]: %s", traceback(tostring(err), 2)); return err; end
8690 0f6623712239 net.http: Allow enabling/disabling error suppression, useful for tests Matthew Wild <mwild1@gmail.com> parents: 8689 diff changeset	44 local function log_if_failed(req, ret, ...)
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	45 if not ret then
10112 b327f2870382 net.: Remove tostring call from logging Kim Alvefur <zash@zash.se>* parents: 9611 diff changeset	46 log("error", "Request '%s': error in callback: %s", req.id, (...));
8690 0f6623712239 net.http: Allow enabling/disabling error suppression, useful for tests Matthew Wild <mwild1@gmail.com> parents: 8689 diff changeset	47 if not req.suppress_errors then
0f6623712239 net.http: Allow enabling/disabling error suppression, useful for tests Matthew Wild <mwild1@gmail.com> parents: 8689 diff changeset	48 error(...);
0f6623712239 net.http: Allow enabling/disabling error suppression, useful for tests Matthew Wild <mwild1@gmail.com> parents: 8689 diff changeset	49 end
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	50 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	51 return ...;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	52 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	53
13319 6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	54 local function destroy_request(request, force)
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	55 local conn = request.conn;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	56 if conn then
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	57 request.conn = nil;
13319 6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	58 local pool = request.http.pool;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	59 if pool and not force then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	60 local pool_id = request.scheme .. "://" .. request.authority;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	61 if not pool[pool_id] then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	62 pool[conn] = pool_id;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	63 pool[pool_id] = conn;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	64 return;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	65 end
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	66 end
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	67 conn:close()
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	68 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	69 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	70
11016 5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	71 local function cancel_request(request, reason)
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	72 if request.callback then
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	73 request.callback(reason or "cancelled", 0, request);
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	74 request.callback = nil;
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	75 end
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	76 if request.conn then
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	77 destroy_request(request);
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	78 end
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	79 end
5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	80
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	81 local function request_reader(request, data, err)
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	82 if not request.parser then
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	83 local function error_cb(reason)
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	84 if request.callback then
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	85 request.callback(reason or "connection-closed", 0, request);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	86 request.callback = nil;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	87 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	88 destroy_request(request);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	89 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	90
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	91 if not data then
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	92 error_cb(err);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	93 return;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	94 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	95
11185 409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	96 local finalize_sink;
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	97 local function success_cb(r)
11185 409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	98 if r.partial then
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	99 -- Request should be streamed
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	100 log("debug", "Request '%s': partial response (%s%s)",
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	101 request.id,
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	102 r.chunked and "chunked, " or "",
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	103 r.body_length and ("%d bytes"):format(r.body_length) or "unknown length"
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	104 );
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	105 if request.streaming_handler then
12881 91baddaeea84 net.http: Add missing log parameter Matthew Wild <mwild1@gmail.com> parents: 12273 diff changeset	106 log("debug", "Request '%s': Streaming via handler", request.id);
11185 409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	107 r.body_sink, finalize_sink = request.streaming_handler(r);
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	108 end
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	109 return;
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	110 elseif finalize_sink then
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	111 log("debug", "Request '%s': Finalizing response stream");
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	112 finalize_sink(r);
409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	113 end
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	114 if request.callback then
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	115 request.callback(r.body, r.code, r, request);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	116 request.callback = nil;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	117 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	118 destroy_request(request);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	119 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	120 local function options_cb()
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	121 return request;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	122 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	123 request.parser = httpstream_new(success_cb, error_cb, "client", options_cb);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	124 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	125 request.parser:feed(data);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	126 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	127
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	128 -- Connection listener callbacks
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	129 function listener.onconnect(conn)
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	130 local req = requests[conn];
8199 8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	131
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	132 -- Initialize request object
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	133 req.write = function (...) return req.conn:write(...); end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	134 local callback = req.callback;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	135 req.callback = function (content, code, response, request)
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	136 do
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	137 local event = { http = req.http, url = req.url, request = req, response = response, content = content, code = code, callback = req.callback };
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	138 req.http.events.fire_event("response", event);
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	139 content, code, response = event.content, event.code, event.response;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	140 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	141
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	142 log("debug", "Request '%s': Calling callback, status %s", req.id, code or "---");
9562 acf74ad0b795 Many things: switch from hacky multi-arg xpcall implementations to a standard util.xpcall Matthew Wild <mwild1@gmail.com> parents: 8731 diff changeset	143 return log_if_failed(req.id, xpcall(callback, handleerr, content, code, response, request));
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	144 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	145 req.reader = request_reader;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	146 req.state = "status";
11016 5176d9f727f6 net.http: Add request:cancel() method Matthew Wild <mwild1@gmail.com> parents: 11015 diff changeset	147 req.cancel = cancel_request;
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	148
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	149 requests[req.conn] = req;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	150
8199 8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	151 -- Validate certificate
8200 e92585ab4998 net.http: Add option for disabling TLS certifictate validation Kim Alvefur <zash@zash.se> parents: 8199 diff changeset	152 if not req.insecure and conn:ssl() then
8199 8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	153 local sock = conn:socket();
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	154 local chain_valid = sock.getpeerverification and sock:getpeerverification();
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	155 if not chain_valid then
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	156 req.callback("certificate-chain-invalid", 0, req);
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	157 req.callback = nil;
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	158 conn:close();
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	159 return;
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	160 end
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	161 local cert = sock.getpeercertificate and sock:getpeercertificate();
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	162 if not cert or not verify_identity(req.host, false, cert) then
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	163 req.callback("certificate-verify-failed", 0, req);
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	164 req.callback = nil;
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	165 conn:close();
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	166 return;
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	167 end
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	168 end
8f82d3cd0631 net.http: Validate HTTPS certificates (fixes #659) Kim Alvefur <zash@zash.se> parents: 8197 diff changeset	169
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	170 -- Send the request
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	171 local request_line = { req.method or "GET", " ", req.path, " HTTP/1.1\r\n" };
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	172 if req.query then
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	173 t_insert(request_line, 4, "?"..req.query);
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	174 end
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	175
11661 735b8f4a6d7e net.http: Send entire HTTP request header as one write Kim Alvefur <zash@zash.se> parents: 11220 diff changeset	176 for k, v in pairs(req.headers) do
735b8f4a6d7e net.http: Send entire HTTP request header as one write Kim Alvefur <zash@zash.se> parents: 11220 diff changeset	177 t_insert(request_line, k .. ": " .. v .. "\r\n");
735b8f4a6d7e net.http: Send entire HTTP request header as one write Kim Alvefur <zash@zash.se> parents: 11220 diff changeset	178 end
735b8f4a6d7e net.http: Send entire HTTP request header as one write Kim Alvefur <zash@zash.se> parents: 11220 diff changeset	179 t_insert(request_line, "\r\n")
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	180 conn:write(t_concat(request_line));
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	181
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	182 if req.body then
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	183 conn:write(req.body);
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	184 end
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	185 end
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	186
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	187 function listener.onincoming(conn, data)
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	188 local request = requests[conn];
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	189
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	190 if not request then
10112 b327f2870382 net.: Remove tostring call from logging Kim Alvefur <zash@zash.se>* parents: 9611 diff changeset	191 log("warn", "Received response from connection %s with no request attached!", conn);
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	192 return;
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	193 end
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	194
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	195 if data and request.reader then
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	196 request:reader(data);
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	197 end
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	198 end
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	199
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	200 function listener.ondisconnect(conn, err)
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	201 local request = requests[conn];
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	202 if request and request.conn then
8045 55a56dc935f2 net.http: Pass error all the way to callback Kim Alvefur <zash@zash.se> parents: 7793 diff changeset	203 request:reader(nil, err or "closed");
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	204 end
13319 6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	205 if request and request.http.pool then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	206 local pool = request.http.pool;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	207 local pool_id = pool[conn];
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	208 if pool_id then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	209 pool[pool_id], pool[conn] = nil, nil;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	210 end
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	211 end
4557 2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	212 requests[conn] = nil;
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	213 end
2abe4e541d52 net.http, httpclient_listener: Merge listener into net.http Matthew Wild <mwild1@gmail.com> parents: 4471 diff changeset	214
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	215 function listener.onattach(conn, req)
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	216 requests[conn] = req;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	217 req.conn = conn;
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	218 end
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	219
6380 4220ffb87b22 net.http, net.http.server, mod_c2s, mod_s2s, mod_component, mod_admin_telnet, mod_net_multiplex: Add ondetach to release connection from 'sessions' table (or equivalent) Matthew Wild <mwild1@gmail.com> parents: 5948 diff changeset	220 function listener.ondetach(conn)
4220ffb87b22 net.http, net.http.server, mod_c2s, mod_s2s, mod_component, mod_admin_telnet, mod_net_multiplex: Add ondetach to release connection from 'sessions' table (or equivalent) Matthew Wild <mwild1@gmail.com> parents: 5948 diff changeset	221 requests[conn] = nil;
4220ffb87b22 net.http, net.http.server, mod_c2s, mod_s2s, mod_component, mod_admin_telnet, mod_net_multiplex: Add ondetach to release connection from 'sessions' table (or equivalent) Matthew Wild <mwild1@gmail.com> parents: 5948 diff changeset	222 end
4220ffb87b22 net.http, net.http.server, mod_c2s, mod_s2s, mod_component, mod_admin_telnet, mod_net_multiplex: Add ondetach to release connection from 'sessions' table (or equivalent) Matthew Wild <mwild1@gmail.com> parents: 5948 diff changeset	223
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	224 function listener.onfail(req, reason)
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	225 req.http.events.fire_event("request-connection-error", { http = req.http, request = req, url = req.url, err = reason });
2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	226 req.callback(reason or "connection failed", 0, req);
7464 3b7de72e58a9 net.http: Add log messages for requests, including their id (so "calling callback" and tracebacks can be traced back to their initial request) Matthew Wild <mwild1@gmail.com> parents: 7463 diff changeset	227 end
3b7de72e58a9 net.http: Add log messages for requests, including their id (so "calling callback" and tracebacks can be traced back to their initial request) Matthew Wild <mwild1@gmail.com> parents: 7463 diff changeset	228
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	229 local function request(self, u, ex, callback)
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	230 local req = url.parse(u);
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	231
903 6737d005a84a net.http: Don't throw error on invalid URLs. Fixes #56. Matthew Wild <mwild1@gmail.com> parents: 739 diff changeset	232 if not (req and req.host) then
8045 55a56dc935f2 net.http: Pass error all the way to callback Kim Alvefur <zash@zash.se> parents: 7793 diff changeset	233 callback("invalid-url", 0, req);
903 6737d005a84a net.http: Don't throw error on invalid URLs. Fixes #56. Matthew Wild <mwild1@gmail.com> parents: 739 diff changeset	234 return nil, "invalid-url";
6737d005a84a net.http: Don't throw error on invalid URLs. Fixes #56. Matthew Wild <mwild1@gmail.com> parents: 739 diff changeset	235 end
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	236
10994 e2ce067bb59a net.http: Fix traceback on invalid URL passed to request() Matthew Wild <mwild1@gmail.com> parents: 9611 diff changeset	237 req.url = u;
e2ce067bb59a net.http: Fix traceback on invalid URL passed to request() Matthew Wild <mwild1@gmail.com> parents: 9611 diff changeset	238 req.http = self;
11220 9b25eecde9e6 net.http: track time of request for debug/stats purposes Matthew Wild <mwild1@gmail.com> parents: 11185 diff changeset	239 req.time = os_time();
10994 e2ce067bb59a net.http: Fix traceback on invalid URL passed to request() Matthew Wild <mwild1@gmail.com> parents: 9611 diff changeset	240
903 6737d005a84a net.http: Don't throw error on invalid URLs. Fixes #56. Matthew Wild <mwild1@gmail.com> parents: 739 diff changeset	241 if not req.path then
6737d005a84a net.http: Don't throw error on invalid URLs. Fixes #56. Matthew Wild <mwild1@gmail.com> parents: 739 diff changeset	242 req.path = "/";
6737d005a84a net.http: Don't throw error on invalid URLs. Fixes #56. Matthew Wild <mwild1@gmail.com> parents: 739 diff changeset	243 end
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	244
7463 3b6e7ce9431f net.http: Add request.id to every request object (can be overridden by providing ex.id) Matthew Wild <mwild1@gmail.com> parents: 6823 diff changeset	245 req.id = ex and ex.id or make_id(req);
3b6e7ce9431f net.http: Add request.id to every request object (can be overridden by providing ex.id) Matthew Wild <mwild1@gmail.com> parents: 6823 diff changeset	246
8114 12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	247 do
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	248 local event = { http = self, url = u, request = req, options = ex, callback = callback };
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	249 local ret = self.events.fire_event("pre-request", event);
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	250 if ret then
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	251 return ret;
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	252 end
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	253 req, u, ex, req.callback = event.request, event.url, event.options, event.callback;
8114 12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	254 end
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	255
4352 912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	256 local method, headers, body;
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	257
5714 520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	258 local host, port = req.host, req.port;
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	259 local host_header = host;
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	260 if (port == "80" and req.scheme == "http")
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	261 or (port == "443" and req.scheme == "https") then
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	262 port = nil;
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	263 elseif port then
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	264 host_header = host_header..":"..port;
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	265 end
520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	266
4352 912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	267 headers = {
5714 520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	268 ["Host"] = host_header;
4352 912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	269 ["User-Agent"] = "Prosody XMPP Server";
912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	270 };
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	271
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	272 if req.userinfo then
4972 1777271a1ec0 net.http: Use base64 from util.encodings instead of luasocket Kim Alvefur <zash@zash.se> parents: 4865 diff changeset	273 headers["Authorization"] = "Basic "..b64(req.userinfo);
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	274 end
4351 3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	275
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	276 if ex then
69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	277 req.onlystatus = ex.onlystatus;
69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	278 body = ex.body;
69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	279 if body then
4369 3578ff5d3674 net.http: Remove extra space after method in request status line for POST. Waqas Hussain <waqas20@gmail.com> parents: 4356 diff changeset	280 method = "POST";
4352 912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	281 headers["Content-Length"] = tostring(#body);
912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	282 headers["Content-Type"] = "application/x-www-form-urlencoded";
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	283 end
4351 3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	284 if ex.method then method = ex.method; end
3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	285 if ex.headers then
3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	286 for k, v in pairs(ex.headers) do
3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	287 headers[k] = v;
3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	288 end
3f414091a008 net.http: Whitespace fixes Matthew Wild <mwild1@gmail.com> parents: 4350 diff changeset	289 end
8200 e92585ab4998 net.http: Add option for disabling TLS certifictate validation Kim Alvefur <zash@zash.se> parents: 8199 diff changeset	290 req.insecure = ex.insecure;
8690 0f6623712239 net.http: Allow enabling/disabling error suppression, useful for tests Matthew Wild <mwild1@gmail.com> parents: 8689 diff changeset	291 req.suppress_errors = ex.suppress_errors;
11185 409ce7686c11 net.http: Add support for streaming chunked/large responses Matthew Wild <mwild1@gmail.com> parents: 11068 diff changeset	292 req.streaming_handler = ex.streaming_handler;
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	293 end
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	294
7520 fc6c24cb3599 net.http: Add quotes around ids in log messages Matthew Wild <mwild1@gmail.com> parents: 7464 diff changeset	295 log("debug", "Making %s %s request '%s' to %s", req.scheme:upper(), method or "GET", req.id, (ex and ex.suppress_url and host_header) or u);
7464 3b7de72e58a9 net.http: Add log messages for requests, including their id (so "calling callback" and tracebacks can be traced back to their initial request) Matthew Wild <mwild1@gmail.com> parents: 7463 diff changeset	296
4352 912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	297 -- Attach to request object
912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	298 req.method, req.headers, req.body = method, headers, body;
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	299
4352 912a49b1c4e3 net.http, httpclient_listener: Move request sending from net.http to onconnect() handler, and add support for HTTPS requests to net.http Matthew Wild <mwild1@gmail.com> parents: 4351 diff changeset	300 local using_https = req.scheme == "https";
5448 cbe9fa2d3787 net.http: Throw error when connecting to a http:// URL without LuaSec available Matthew Wild <mwild1@gmail.com> parents: 5354 diff changeset	301 if using_https and not ssl_available then
cbe9fa2d3787 net.http: Throw error when connecting to a http:// URL without LuaSec available Matthew Wild <mwild1@gmail.com> parents: 5354 diff changeset	302 error("SSL not available, unable to contact https URL");
cbe9fa2d3787 net.http: Throw error when connecting to a http:// URL without LuaSec available Matthew Wild <mwild1@gmail.com> parents: 5354 diff changeset	303 end
5714 520671c3159c net.http: Include port number (when non-standard) in the Host header of outgoing requests, as per the HTTP RFC Matthew Wild <mwild1@gmail.com> parents: 5505 diff changeset	304 local port_number = port and tonumber(port) or (using_https and 443 or 80);
5776 bd0ff8ae98a8 Remove all trailing whitespace Florian Zeitz <florob@babelmonkeys.de> parents: 5714 diff changeset	305
12273 c0f49a4026f8 net.http: Allow using DANE via options or per request settings Kim Alvefur <zash@zash.se> parents: 11749 diff changeset	306 local use_dane = self.options and self.options.use_dane;
5353 8c3f28f5c1c1 net.http: Allow passing an SSL context or options table to be used for HTTPS requests (thanks daurnimator) Matthew Wild <mwild1@gmail.com> parents: 4977 diff changeset	307 local sslctx = false;
8c3f28f5c1c1 net.http: Allow passing an SSL context or options table to be used for HTTPS requests (thanks daurnimator) Matthew Wild <mwild1@gmail.com> parents: 4977 diff changeset	308 if using_https then
8197 55826e29c719 net.http: Move default SSL/TLS settings into options, allowing them to be overriden in new() Kim Alvefur <zash@zash.se> parents: 8196 diff changeset	309 sslctx = ex and ex.sslctx or self.options and self.options.sslctx;
12273 c0f49a4026f8 net.http: Allow using DANE via options or per request settings Kim Alvefur <zash@zash.se> parents: 11749 diff changeset	310 if ex and ex.use_dane ~= nil then
c0f49a4026f8 net.http: Allow using DANE via options or per request settings Kim Alvefur <zash@zash.se> parents: 11749 diff changeset	311 use_dane = ex.use_dane;
c0f49a4026f8 net.http: Allow using DANE via options or per request settings Kim Alvefur <zash@zash.se> parents: 11749 diff changeset	312 end
5353 8c3f28f5c1c1 net.http: Allow passing an SSL context or options table to be used for HTTPS requests (thanks daurnimator) Matthew Wild <mwild1@gmail.com> parents: 4977 diff changeset	313 end
8c3f28f5c1c1 net.http: Allow passing an SSL context or options table to be used for HTTPS requests (thanks daurnimator) Matthew Wild <mwild1@gmail.com> parents: 4977 diff changeset	314
13319 6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	315 if self.pool then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	316 local pool_id = req.scheme .. "://" .. req.authority;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	317 local conn = self.pool[pool_id];
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	318 if conn then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	319 log("debug", "Re-using connection to %s from pool", req.host);
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	320 self.pool[pool_id] = nil;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	321 self.pool[conn] = nil;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	322 req.conn = conn;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	323 requests[conn] = req;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	324 self.events.fire_event("request", { http = self, request = req, url = u });
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	325 listener.onconnect(conn);
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	326 return req;
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	327 else
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	328 log("debug", "Opening a new connection for this request");
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	329 end
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	330 end
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	331
12273 c0f49a4026f8 net.http: Allow using DANE via options or per request settings Kim Alvefur <zash@zash.se> parents: 11749 diff changeset	332 local http_service = basic_resolver.new(host, port_number, "tcp", { servername = req.host; use_dane = use_dane });
8551 2bd2e94a0496 net.http: Refactor to use new net.connect API, brings support for async DNS Matthew Wild <mwild1@gmail.com> parents: 8535 diff changeset	333 connect(http_service, listener, { sslctx = sslctx }, req);
8114 12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	334
12df41a5a4b1 net.http: Fire new events: pre-request, request-connection-error, request, response Matthew Wild <mwild1@gmail.com> parents: 8113 diff changeset	335 self.events.fire_event("request", { http = self, request = req, url = u });
616 69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	336 return req;
69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	337 end
69bc5782b25e Non-blocking HTTP requests (adding net.http) Matthew Wild <mwild1@gmail.com> parents: diff changeset	338
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	339 local function new(options)
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	340 local http = {
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	341 options = options;
10803 71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	342 request = function (self, u, ex, callback)
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	343 if callback ~= nil then
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	344 return request(self, u, ex, callback);
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	345 else
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	346 return promise.new(function (resolve, reject)
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	347 request(self, u, ex, function (body, code, a, b)
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	348 if code == 0 then
11048 160308b4b384 net.http: use new net.http.errors lib for creating error object Matthew Wild <mwild1@gmail.com> parents: 11017 diff changeset	349 reject(http_errors.new(body, { request = a }));
10803 71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	350 else
11049 f103f59ea2b5 net.http: http.request() promise now resolves with response (breaking change) Matthew Wild <mwild1@gmail.com> parents: 11048 diff changeset	351 a.request = b;
f103f59ea2b5 net.http: http.request() promise now resolves with response (breaking change) Matthew Wild <mwild1@gmail.com> parents: 11048 diff changeset	352 resolve(a);
10803 71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	353 end
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	354 end);
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	355 end);
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	356 end
71d04bd6cadd net.http: Return a Promise if no callback is given Kim Alvefur <zash@zash.se> parents: 10464 diff changeset	357 end;
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	358 new = options and function (new_options)
9611 2700317f93e4 net.http: Manually merge settings (fixes #1231) Kim Alvefur <zash@zash.se> parents: 9562 diff changeset	359 local final_options = {};
2700317f93e4 net.http: Manually merge settings (fixes #1231) Kim Alvefur <zash@zash.se> parents: 9562 diff changeset	360 for k, v in pairs(options) do final_options[k] = v; end
2700317f93e4 net.http: Manually merge settings (fixes #1231) Kim Alvefur <zash@zash.se> parents: 9562 diff changeset	361 if new_options then
2700317f93e4 net.http: Manually merge settings (fixes #1231) Kim Alvefur <zash@zash.se> parents: 9562 diff changeset	362 for k, v in pairs(new_options) do final_options[k] = v; end
2700317f93e4 net.http: Manually merge settings (fixes #1231) Kim Alvefur <zash@zash.se> parents: 9562 diff changeset	363 end
2700317f93e4 net.http: Manually merge settings (fixes #1231) Kim Alvefur <zash@zash.se> parents: 9562 diff changeset	364 return new(final_options);
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	365 end or new;
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	366 events = events.new();
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	367 };
13319 6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	368 if options and options.connection_pooling then
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	369 -- util.cache in the future?
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	370 http.pool = {};
6d6291dfe735 net.http: Add simple connection pooling Kim Alvefur <zash@zash.se> parents: 12974 diff changeset	371 end
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	372 return http;
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	373 end
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	374
8197 55826e29c719 net.http: Move default SSL/TLS settings into options, allowing them to be overriden in new() Kim Alvefur <zash@zash.se> parents: 8196 diff changeset	375 local default_http = new({
11749 83d6d6a70edf net.http: fail open if surrounding code does not configure TLS Jonas Schäfer <jonas@wielicki.name> parents: 11661 diff changeset	376 sslctx = { mode = "client", protocol = "sslv23", options = { "no_sslv2", "no_sslv3" }, alpn = "http/1.1", verify = "peer" };
8690 0f6623712239 net.http: Allow enabling/disabling error suppression, useful for tests Matthew Wild <mwild1@gmail.com> parents: 8689 diff changeset	377 suppress_errors = true;
8197 55826e29c719 net.http: Move default SSL/TLS settings into options, allowing them to be overriden in new() Kim Alvefur <zash@zash.se> parents: 8196 diff changeset	378 });
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	379
6780 647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	380 return {
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	381 request = function (u, ex, callback)
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	382 return default_http:request(u, ex, callback);
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	383 end;
8196 bc2bcfa63b43 net.http: Expose defaults Kim Alvefur <zash@zash.se> parents: 8195 diff changeset	384 default = default_http;
8113 cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	385 new = new;
cfb5ab763384 net.http: Allow creation of http client objects, with custom options Matthew Wild <mwild1@gmail.com> parents: 8045 diff changeset	386 events = default_http.events;
6780 647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	387 -- COMPAT
647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	388 urlencode = util_http.urlencode;
647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	389 urldecode = util_http.urldecode;
647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	390 formencode = util_http.formencode;
647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	391 formdecode = util_http.formdecode;
11015 355eae2f9ba8 net.http: Re-expose destroy_request() function Matthew Wild <mwild1@gmail.com> parents: 10994 diff changeset	392 destroy_request = destroy_request;
11067 f2ffc16a9669 net.http: Add feature discovery (currently just contains SNI) Matthew Wild <mwild1@gmail.com> parents: 11063 diff changeset	393 features = {
f2ffc16a9669 net.http: Add feature discovery (currently just contains SNI) Matthew Wild <mwild1@gmail.com> parents: 11063 diff changeset	394 sni = true;
f2ffc16a9669 net.http: Add feature discovery (currently just contains SNI) Matthew Wild <mwild1@gmail.com> parents: 11063 diff changeset	395 };
6780 647adfd8f738 net.: Remove use of module() function Kim Alvefur <zash@zash.se>* parents: 6501 diff changeset	396 };

Mercurial > prosody-hg

annotate net/http.lua @ 13319:6d6291dfe735