about summary refs log tree commit diff
path: root/app/lib/request.rb
AgeCommit message (Collapse)Author
2022-09-20Improve error reporting and logging when processing remote accounts (#15605)Claire
* Add a more descriptive PrivateNetworkAddressError exception class * Remove unnecessary exception class to rescue clause * Remove unnecessary include to JsonLdHelper * Give more neutral error message when too many webfinger redirects * Remove unnecessary guard condition * Rework how “ActivityPub::FetchRemoteAccountService” handles errors Add “suppress_errors” keyword argument to avoid raising errors in ActivityPub::FetchRemoteAccountService#call (default/previous behavior). * Rework how “ActivityPub::FetchRemoteKeyService” handles errors Add “suppress_errors” keyword argument to avoid raising errors in ActivityPub::FetchRemoteKeyService#call (default/previous behavior). * Fix Webfinger::RedirectError not being a subclass of Webfinger::Error * Add suppress_errors option to ResolveAccountService Defaults to true (to preserve previous behavior). If set to false, errors will be raised instead of caught, allowing the caller to be informed of what went wrong. * Return more precise error when failing to fetch account signing AP payloads * Add tests * Fixes * Refactor error handling a bit * Fix various issues * Add specific error when provided Digest is not 256 bits of base64-encoded data * Please CodeClimate * Improve webfinger error reporting
2022-08-25Support "http_hidden_proxy" ENV var for hidden service only proxy (#18427)Jeong Arm
* Support "http_hidden_proxy" ENV var for hidden service only proxy * Fallback to http_proxy if http_hidden_proxy is not set
2021-11-05Add support for structured data and more OpenGraph tags to link cards (#16938)Eugen Rochko
Save preview cards under their canonical URL Increase max redirects to follow from 2 to 3
2021-01-22Optimize some regex matching (#15528)luigi
* Use Regex#match? * Replace =~ too * Avoid to call match? from Nil * Keep value of Regexp.last_match
2020-09-26Add environment variable to allow requests to some private addresses (#14722)ThibG
2020-09-01Bump rubocop from 0.86.0 to 0.88.0 (#14412)dependabot[bot]
* Bump rubocop from 0.86.0 to 0.88.0 Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0. - [Release notes](https://github.com/rubocop-hq/rubocop/releases) - [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0) Signed-off-by: dependabot[bot] <support@github.com> * Fix for latest RuboCop Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-07-14Fix rubocop warning (#14288)abcang
* Fix rubocop warning * use limit variable * use ContextCreatingMethods option
2020-05-10Fix error within error when limiting backtrace to 3 lines (#13120)Eugen Rochko
Fix #13086, close #13113
2020-01-12Fix Ruby 2.7 support (#12831)ThibG
- update http gem to avoid errors - update blurhash gem to avoid shared object loading error - update goldfinger gem so the http gem could be updated - update json gem to avoid warnings
2020-01-11Suppress backtrace when delivering toots (#12798)Daigo 3 Dango
This is to suppress irrelevant backtrace from errors raised when delivering toots to remote servers. The errors are usually out of control by the local server and backtraces don't provide much information. This is similar to https://github.com/tootsuite/mastodon/pull/5174 and shortens backtraces like below: ``` WARN: Mastodon::UnexpectedResponseError: https://example.com/inbox returned code 523 WARN: app/workers/activitypub/delivery_worker.rb:48:in `block (3 levels) in perform_request' app/lib/request.rb:75:in `perform' app/workers/activitypub/delivery_worker.rb:47:in `block (2 levels) in perform_request' app/lib/request_pool.rb:53:in `use' app/lib/request_pool.rb:108:in `block (2 levels) in with' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/notifications.rb:170:in `instrument' app/lib/request_pool.rb:107:in `block in with' app/lib/connection_pool/shared_connection_pool.rb:21:in `block (2 levels) in with' app/lib/connection_pool/shared_connection_pool.rb:20:in `handle_interrupt' app/lib/connection_pool/shared_connection_pool.rb:20:in `block in with' app/lib/connection_pool/shared_connection_pool.rb:16:in `handle_interrupt' app/lib/connection_pool/shared_connection_pool.rb:16:in `with' app/lib/request_pool.rb:106:in `with' app/workers/activitypub/delivery_worker.rb:46:in `block in perform_request' vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:51:in `run_code' vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:42:in `run_yellow' vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:24:in `run' app/workers/activitypub/delivery_worker.rb:57:in `perform_request' app/workers/activitypub/delivery_worker.rb:25:in `perform' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call' vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time' vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' app/lib/sidekiq_error_handler.rb:5:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread' ``` ``` WARN: Stoplight::Error::RedLight: https://example.com/inbox WARN: vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:46:in `run_red' vendor/bundle/ruby/2.7.0/gems/stoplight-2.2.0/lib/stoplight/light/runnable.rb:25:in `run' app/workers/activitypub/delivery_worker.rb:57:in `perform_request' app/workers/activitypub/delivery_worker.rb:25:in `perform' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:31:in `block in call' vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/statsd/publisher.rb:27:in `statsd_time' vendor/bundle/ruby/2.7.0/gems/nsa-0.2.7/lib/nsa/collectors/sidekiq.rb:30:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' app/lib/sidekiq_error_handler.rb:5:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/scout_apm-2.3.0.pre3/lib/scout_apm/background_job_integrations/sidekiq.rb:69:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/sidekiq-unique-jobs-6.0.18/lib/sidekiq_unique_jobs/server/middleware.rb:29:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:73:in `block in wrap' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/execution_wrapper.rb:87:in `wrap' vendor/bundle/ruby/2.7.0/gems/activesupport-5.2.4.1/lib/active_support/reloader.rb:72:in `wrap' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog' vendor/bundle/ruby/2.7.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread' ```
2019-09-05Attempt to concurrently connect to remote IP addresses (#11757)ThibG
* Attempt to concurrently connect to remote IP addresses * Reduce code length to please CodeClimate 🤷
2019-07-11Add HTTP signatures to all outgoing ActivityPub GET requests (#11284)Eugen Rochko
2019-07-10Refactor fetching of remote resources (#11251)Eugen Rochko
2019-07-07Fix URLs appearing twice in errors of ActivityPub::DeliveryWorker (#11231)Eugen Rochko
2019-07-07Fix support for HTTP proxies (#11245)ThibG
* Disable incorrect check for hidden services in Socket Hidden services can only be accessed with an HTTP proxy, in which case the host seen by the Socket class will be the proxy, not the target host. Hidden services are already filtered in `Request#initialize`. * Use our Socket class to connect to HTTP proxies Avoid the timeout logic being bypassed * Add support for IP addresses in Request::Socket * Refactor a bit, no need to keep the DNS resolver around
2019-07-05Fix HTTP requests to IPv6 hosts (#11240)ThibG
2019-07-02Add request pool to improve delivery performance (#10353)Eugen Rochko
* Add request pool to improve delivery performance Fix #7909 * Ensure connection is closed when exception interrupts execution * Remove Timeout#timeout from socket connection * Fix infinite retrial loop on HTTP::ConnectionError * Close sockets on failure, reduce idle time to 90 seconds * Add MAX_REQUEST_POOL_SIZE option to limit concurrent connections to the same server * Use a shared pool size, 512 by default, to stay below open file limit * Add some tests * Add more tests * Reduce MAX_IDLE_TIME from 90 to 30 seconds, reap every 30 seconds * Use a shared pool that returns preferred connection but re-purposes other ones when needed * Fix wrong connection being returned on subsequent calls within the same thread * Reduce mutex calls on flushes from 2 to 1 and add test for reaping
2019-03-11Increase DNS timeouts (#10238)ThibG
2018-12-26Fix ThreadResolveWorker getting queued with invalid URLs (#9628)Eugen Rochko
2018-11-27Fix TLS handshake timeout not being enforced (#9381)Eugen Rochko
Follow-up to #9329
2018-11-27Fix nil error when no DNS addresses are found for host (#9379)Eugen Rochko
2018-11-22Fix connect timeout not being enforced (#9329)Eugen Rochko
* Fix connect timeout not being enforced The loop was catching the timeout exception that should stop execution, so the next IP would no longer be within a timed block, which led to requests taking much longer than 10 seconds. * Use timeout on each IP attempt, but limit to 2 attempts * Fix code style issue * Do not break Request#perform if no block given * Update method stub in spec for Request * Move timeout inside the begin/rescue block * Use Resolv::DNS with timeout of 1 to get IP addresses * Update Request spec to stub Resolv::DNS instead of Addrinfo * Fix Resolve::DNS stubs in Request spec
2018-11-21Revert connect timeout from 1s to 10s (#9319)Eugen Rochko
The failure rate in Sidekiq is too high
2018-11-08Reduce connect timeout limit and limit signature failures by source IP (#9236)Eugen Rochko
* Reduce connect timeout from 10s to 1s * Limit failing signature verifications per source IP
2018-08-31Do not sign useless User-Agent or Accept-Encoding headers (#8533)Eugen Rochko
Fix #8080
2018-08-26Add CLI task for rotating keys (#8466)Eugen Rochko
* If an Update is signed with known key, skip re-following procedure Because it means the remote actor did *not* lose their database * Add CLI method for rotating keys bin/tootctl accounts rotate [USERNAME] Generates a new RSA key per account and sends out an Update activity signed with the old key. * Key rotation: Space out Update fan-outs every 5 minutes per 1000 accounts * Skip suspended accounts in key rotation
2018-06-29Merge `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into ↵MIYAGI Hikaru
`ALLOW_ACCESS_TO_HIDDEN_SERVICE` (#7901) If Mastodon accesses to the hidden service via transparent proxy, it's needed to avoid checking whether it's a private address, since `.onion` is resolved to a private address. I was previously using the `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` to provide that function. However, I realized that using `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` is redundant, since this specification is always used with `ALLOW_ACCESS_TO_HIDDEN_SERVICE`. Therefore, I decided to integrate the setting of `HIDDEN_SERVICE_VIA_TRANSPARENT_PROXY` into` ALLOW_ACCESS_TO_HIDDEN_SERVICE`.
2018-05-18User agent for WebFinger (#7531)MIYAGI Hikaru
* User agent for WebFinger * local_domain → web_domain * 'http' is away accidentally...
2018-05-10Disable inflate gzip on head method (#7432)Yamagishi Kazutoshi
2018-05-09Support gzip encoding on Request (#7425)Yamagishi Kazutoshi
2018-05-02Guard against nil URLs in Request class (#7284)Eugen Rochko
Fix #7265
2018-04-25HTTP proxy support for outgoing request, manage access to hidden service (#7134)MIYAGI Hikaru
* Add support for HTTP client proxy * Add access control for darknet Supress error when access to darknet via transparent proxy * Fix the codes pointed out * Lint * Fix an omission + lint * any? -> include? * Change detection method to regexp to avoid test fail
2018-03-26Validate HTTP response length while receiving (#6891)Akihiko Odaki
to_s method of HTTP::Response keeps blocking while it receives the whole content, no matter how it is big. This means it may waste time to receive unacceptably large files. It may also consume memory and disk in the process. This solves the inefficency by checking response length while receiving.
2018-03-24Close http connection in perform method of Request class (#6889)Akihiko Odaki
HTTP connections must be explicitly closed in many cases, and letting perform method close connections makes its callers less redundant and prevent them from forgetting to close connections.
2018-03-20request: in the event of failure, try other IPs (#6761) (#6813)Rey Tucker
* request: in the event of failure, try other IPs (#6761) In the case where a name has multiple A/AAAA records, we should try subsequent records instead of immediately failing when we have a failure on the first IP address. This significantly improves delivery success when there are network connectivity problems affecting only IPv4 or IPv6. * fix method call style * request_spec: adjust test case to use Addrinfo * request: Request/open: move private addr check to within begin/rescue * request_spec: add case to test failover, fix exception check * Double Addrinfo.foreach so that it correctly yields instances
2018-02-24Raise Mastodon::HostValidationError when host for HTTP request is private ↵Akihiko Odaki
(#6410)
2017-12-06Using double splat operator (#5859)Yamagishi Kazutoshi
2017-10-02Suppress backtrace from Request#perform (#5174)Daigo 3 Dango
2017-09-28Change max redirects followed to 2 (#5136)Eugen Rochko
I see no reason to allow more than that. Usually a redirect is HTTP->HTTPS, then maybe URL structure changed, but more than that is highly unlikely to be a legitimate use case.
2017-09-14Include requested URL into the message on network errors (#4945)unarist
2017-08-09Add Digest header to requests with body, handle acct and URI keyId (#4565)Eugen Rochko
2017-07-14HTTP signatures (#4146)Eugen Rochko
* Add Request class with HTTP signature generator Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06 * Add HTTP signature verification concern * Add test for SignatureVerification concern * Add basic test for Request class * Make PuSH subscribe/unsubscribe requests use new Request class Accidentally fix lease_seconds not being set and sent properly, and change the new minimum subscription duration to 1 day * Make all PuSH workers use new Request class * Make Salmon sender use new Request class * Make FetchLinkService use new Request class * Make FetchAtomService use the new Request class * Make Remotable use the new Request class * Make ResolveRemoteAccountService use the new Request class * Add more tests * Allow +-30 seconds window for signed request to remain valid * Disable time window validation for signed requests, restore 7 days as PuSH subscription duration (which was previous default due to a bug)
generated by cgit-pink (git 2.37.1) at 2024-11-09 03:37:16 +0000