about summary refs log tree commit diff
path: root/app/workers
AgeCommit message (Collapse)Author
2017-10-08Set snowflake IDs for backdated statuses (#5260)Eugen Rochko
- Rename Mastodon::TimestampIds into Mastodon::Snowflake for clarity - Skip for statuses coming from inbox, aka delivered in real-time - Skip for statuses that claim to be from the future
2017-10-06Improve error handling on LinkCrawlWorker (#5250)unarist
* Improve error handling on LinkCrawlWorker * Ignore TimeoutError and InvalidURIError too * Record errors to debug log * Enable dead job queue on LinkCrawlWorker Since most of acceptable errors were already ignored, only our side issue should go to dead job queue. * Ignore all http gem errors
2017-10-04Use separate workers to process imports, retry failures (#5207)Eugen Rochko
2017-09-30Retry ActivityPub delivery a few more times (#5014)ThibG
2017-09-29After 7 days of repeated delivery failures, give up on inbox (#5131)Eugen Rochko
- A successful delivery cancels it out - An incoming delivery from account of the inbox cancels it out
2017-09-28Flush body when POST requests (#5128)abcang
2017-09-26Follow-up to #4582 and #5027, removing dead code (#5101)Eugen Rochko
2017-09-24Suppress backtrace when failed to communicate with a remote instance (#5076)Daigo 3 Dango
2017-09-16Fix an error in ReplyDistributionWorker when replied status was deleted (#4974)unarist
Reply distribution is proceed by Sidekiq, so replied status may be deleted before this.
2017-09-14Add scheduled worker to purge old user IPs (#4951)sdukhovni
* Add scheduled worker to purge old user IPs * Use ruby 1.9 hash syntax
2017-09-14Revert unique retry job (#4937)abcang
* Revert "Enable UniqueRetryJobMiddleware even when called from sidekiq worker (#4836)" This reverts commit 6859d4c0289e767955aac3f345074220fe200604. * Revert "Do not execute the job with the same arguments as the retry job (#4814)" This reverts commit be7ffa2d7539d5a1946a3933cb9d242b9fac0ddc.
2017-09-14Fix refollowing (#4931)ThibG
* Make RefollowWorker ActivityPub-only to avoid potential identifier mismatches * Don't call RefollowWorker on new accounts
2017-09-12[WiP] Whenever a remote keypair changes, unfollow them and re-subscribe to ↵ThibG
… (#4907) * Whenever a remote keypair changes, unfollow them and re-subscribe to them In Mastodon (it could be different for other OStatus or AP-enabled software), a keypair change is indicative of whole user (or instance) data loss. In this situation, the “new” user might be different, and almost certainly has an empty followers list. In this case, Mastodon instances will disagree on follower lists, leading to unreliable delivery and “shadow followers”, that is users believed by a remote instance to be followers, without the affected user knowing. Drawbacks of this change are: 1. If an user legitimately changes public key for some reason without losing data (not possible in Mastodon at the moment), they will have their remote followers unsubscribed/re-subscribed needlessly. 2. Depending of the number of remote followers, this may generate quite some traffic. 3. If the user change is an attempt at usurpation, the remote followers will unknowingly follow the usurper. Note that this is *not* a change of behavior, Mastodon already behaves like that, although delivery might be unreliable, and the usurper would not have known the former user's followers. * Rename ResubscribeWorker to RefollowWorker * Process followers in batches
2017-09-10Default follows for new users (#4871)Eugen Rochko
When a new user confirms their e-mail, bootstrap their home timeline by automatically following a set of accounts. By default, all local admin accounts (that are unlocked). Can be customized by new admin setting (comma-separated usernames, local and unlocked only)
2017-09-06Switch to static URIs, new URI format in both protocols for new statuses (#4815)Eugen Rochko
* Decouple Status#local? from uri being nil * Replace on-the-fly URI generation with stored URIs - Generate URI in after_save hook for local statuses - Use static value in TagManager when available, fallback to tag format - Make TagManager use ActivityPub::TagManager to understand new format - Adjust tests * Use other heuristic for locality of old statuses, do not perform long query * Exclude tombstone stream entries from Atom feed * Prevent nil statuses from landing in Pubsubhubbub::DistributionWorker * Fix URI not being saved (#4818) * Add more specs for Status * Save generated uri immediately and also fix method order to minimize diff. * Fix alternate HTML URL in Atom * Fix tests * Remove not-null constraint from statuses migration to speed it up
2017-09-05Do not execute the job with the same arguments as the retry job (#4814)abcang
2017-09-03Instantly upgrade account to ActivityPub if we receive ActivityPub payload ↵Eugen Rochko
(#4766)
2017-08-30Forward ActivityPub creates that reply to local statuses (#4709)Eugen Rochko
* Forward ActivityPub creates that reply to local statuses * Fix test * Fix wrong signers
2017-08-29Serialize ActivityPub alternate link into OStatus deletes, handle it (#4730)Eugen Rochko
Requires moving Atom rendering from DistributionWorker (where `stream_entry.status` is already nil) to inline (where `stream_entry.status.destroyed?` is true) and distributing that. Unfortunately, such XML renderings can no longer be easily chained together into one payload of n items.
2017-08-26Forward ActivityPub deletes to followers of rebloggers (#4706)Eugen Rochko
2017-08-26Add handling of Linked Data Signatures in payloads (#4687)Eugen Rochko
* Add handling of Linked Data Signatures in payloads * Add a way to sign JSON, fix canonicalization of signature options * Fix signatureValue encoding, send out signed JSON when distributing * Add missing security context
2017-08-26authorize-follow-requests-after-unlocking (#4658)masarakki
2017-08-24Add configuration to disable private status federation over PuSH (#4582)Eugen Rochko
2017-08-21Periodically remove expired PuSH subscribers (#4654)Eugen Rochko
2017-08-21ActivityPub migration procedure (#4617)Eugen Rochko
* ActivityPub migration procedure Once one account is detected as going from OStatus to ActivityPub, invalidate WebFinger cache for other accounts from the same domain * Unsubscribe from PuSH updates once we receive an ActivityPub payload * Re-subscribe to PuSH unless already unsubscribed, regardless of protocol
2017-08-19Improve ActivityPub/OStatus compatibility (#4632)Eugen Rochko
*Note: OStatus URIs are invalid for ActivityPub. But we have them for as long as we want to keep old OStatus-sourced content and as long as we remain OStatus-compatible.* - In Announce handling, if object URI is not a URL, fallback to object URL - Do not use specialized ThreadResolveWorker, rely on generalized handling - When serializing notes, if parent's URI is not a URL, use parent's URL
2017-08-13ActivityPub delivery (#4566)Eugen Rochko
* Deliver ActivityPub Like * Deliver ActivityPub Undo-Like * Deliver ActivityPub Create/Announce activities * Deliver ActivityPub creates from mentions * Deliver ActivityPub Block/Undo-Block * Deliver ActivityPub Accept/Reject-Follow * Deliver ActivityPub Undo-Follow * Deliver ActivityPub Follow * Deliver ActivityPub Delete activities Incidentally fix #889 * Adjust BatchedRemoveStatusService for ActivityPub * Add tests for ActivityPub workers * Add tests for FollowService * Add tests for FavouriteService, UnfollowService and PostStatusService * Add tests for ReblogService, BlockService, UnblockService, ProcessMentionsService * Add tests for AuthorizeFollowService, RejectFollowService, RemoveStatusService * Add tests for BatchedRemoveStatusService * Deliver updates to a local account to ActivityPub followers * Minor adjustments
2017-08-12Only PuSH-resubscribe to OStatus accounts (#4583)Eugen Rochko
2017-08-08Add ActivityPub inbox (#4216)Eugen Rochko
* Add ActivityPub inbox * Handle ActivityPub deletes * Handle ActivityPub creates * Handle ActivityPub announces * Stubs for handling all activities that need to be handled * Add ActivityPub actor resolving * Handle conversation URI passing in ActivityPub * Handle content language in ActivityPub * Send accept header when fetching actor, handle JSON parse errors * Test for ActivityPub::FetchRemoteAccountService * Handle public key and icon/image when embedded/as array/as resolvable URI * Implement ActivityPub::FetchRemoteStatusService * Add stubs for more interactions * Undo activities implemented * Handle out of order activities * Hook up ActivityPub to ResolveRemoteAccountService, handle Update Account activities * Add fragment IDs to all transient activity serializers * Add tests and fixes * Add stubs for missing tests * Add more tests * Add more tests
2017-07-31Change RuboCop rules to loose (#4464)Yamagishi Kazutoshi
2017-07-28Fix guard clause in WebPushNotificationWorker (#4421)Eugen Rochko
2017-07-28When PuSH subscribe attempts are exhausted, unsubscribe (#4422)Eugen Rochko
2017-07-27Add callback_url/acct information for Sidekiq PuSH workers Exception. (#4281)Clworld
* Add destination informations to exception on SubscribeWorker and DeliveryWorker. * Simplify delivery error message. * Prevent changing Exception type... * fix typo.
2017-07-26fix(web_push_notification_worker): Guard against deleted notifications (#4379)Sorin Davidoi
2017-07-24Move clean up of unconfirmed users to sidekiq-scheduler (#4336)Eugen Rochko
* Move clean up of unconfirmed users to sidekiq-scheduler * mastodon:daily is now deprecated
2017-07-20Fix webfinger retries (#4275)Eugen Rochko
* Do not raise unretryable exceptions in ResolveRemoteAccountService * Removed fatal exceptions from ResolveRemoteAccountService Exceptions that cannot be retried should not be raised. New exception class for those that can be retried (Mastodon::UnexpectedResponseError)
2017-07-19Correct OStatus inflection (Ostatus -> OStatus) (#4255)Eugen Rochko
2017-07-18Introduce Ostatus name space (#4164)Akihiko Odaki
* Wrap methods of ProcessFeedService::ProcessEntry in classes This is a change same with 425acecfdb15093a265b191120fb2d4e4c4135c4, except that it has the following changes: * Revert irrelevant change in find_or_create_conversation * Fix error handling for RemoteActivity * Introduce Ostatus name space
2017-07-18fix(push-subscriptions): Refactor how Sidekiq jobs are handled (#4226)Sorin Davidoi
2017-07-17Improve a query in WebPushNotificationWorker (#4234)Akihiko Odaki
2017-07-17Use update method to update session in WebPushNotificationWorker (#4235)Akihiko Odaki
2017-07-15Optimize uri normalization (#4212)ThibG
* Add dependency on idn-ruby to speed up URI normalization * Use normalized_host instead of normalize.host when applicable When we are only interested in the normalized host, calling normalized_host avoids normalizing the other components of the URI as well as creating a new object
2017-07-14Fix #2672 - Connect signed PuSH subscription requests to instance domain (#4205)Eugen Rochko
* Fix #2672 - Connect signed PuSH subscription requests to instance domain Resolves #2739 * Fix return of locate_subscription * Fix tests
2017-07-14HTTP signatures (#4146)Eugen Rochko
* Add Request class with HTTP signature generator Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06 * Add HTTP signature verification concern * Add test for SignatureVerification concern * Add basic test for Request class * Make PuSH subscribe/unsubscribe requests use new Request class Accidentally fix lease_seconds not being set and sent properly, and change the new minimum subscription duration to 1 day * Make all PuSH workers use new Request class * Make Salmon sender use new Request class * Make FetchLinkService use new Request class * Make FetchAtomService use the new Request class * Make Remotable use the new Request class * Make ResolveRemoteAccountService use the new Request class * Add more tests * Allow +-30 seconds window for signed request to remain valid * Disable time window validation for signed requests, restore 7 days as PuSH subscription duration (which was previous default due to a bug)
2017-07-13Web Push Notifications (#3243)Sorin Davidoi
* feat: Register push subscription * feat: Notify when mentioned * feat: Boost, favourite, reply, follow, follow request * feat: Notification interaction * feat: Handle change of public key * feat: Unsubscribe if things go wrong * feat: Do not send normal notifications if push is enabled * feat: Focus client if open * refactor: Move push logic to WebPushSubscription * feat: Better title and body * feat: Localize messages * chore: Fix lint errors * feat: Settings * refactor: Lazy load * fix: Check if push settings exist * feat: Device-based preferences * refactor: Simplify logic * refactor: Pull request feedback * refactor: Pull request feedback * refactor: Create /api/web/push_subscriptions endpoint * feat: Spec PushSubscriptionController * refactor: WebPushSubscription => Web::PushSubscription * feat: Spec Web::PushSubscription * feat: Display first media attachment * feat: Support direction * fix: Stuff broken while rebasing * refactor: Integration with session activations * refactor: Cleanup * refactor: Simplify implementation * feat: Set VAPID keys via environment * chore: Comments * fix: Crash when no alerts * fix: Set VAPID keys in testing environment * fix: Follow link * feat: Notification actions * fix: Delete previous subscription * chore: Temporary logs * refactor: Move migration to a later date * fix: Fetch the correct session activation and misc bugs * refactor: Move migration to a later date * fix: Remove follow request (no notifications) * feat: Send administrator contact to push service * feat: Set time-to-live * fix: Do not show sensitive images * fix: Reducer crash in error handling * feat: Add badge * chore: Fix lint error * fix: Checkbox label overlap * fix: Check for payload support * fix: Rename action "type" (crash in latest Chrome) * feat: Action to expand notification * fix: Lint errors * fix: Unescape notification body * fix: Do not allow boosting if the status is hidden * feat: Add VAPID keys to the production sample environment * fix: Strip HTML tags from status * refactor: Better error messages * refactor: Handle browser not implementing the VAPID protocol (Samsung Internet) * fix: Error when target_status is nil * fix: Handle lack of image * fix: Delete reference to invalid subscriptions * feat: Better error handling * fix: Unescape HTML characters after tags are striped * refactor: Simpify code * fix: Modify to work with #4091 * Sort strings alphabetically * i18n: Updated Polish translation it annoys me that it's not fully localized :P * refactor: Use current_session in PushSubscriptionController * fix: Rebase mistake * fix: Set cacheName to mastodon * refactor: Pull request feedback * refactor: Remove logging statements * chore(yarn): Fix conflicts with master * chore(yarn): Copy latest from master * chore(yarn): Readd offline-plugin * refactor: Use save! and update! * refactor: Send notifications async * fix: Allow retry when push fails * fix: Save track for failed pushes * fix: Minify sw.js * fix: Remove account_id from fabricator
2017-07-10Limit total subscribe retries to 10, but space them out more (#4142)Eugen Rochko
Since there is little point in retrying so often when a service is down or does not exist anymore. Subscriptions are renewed 1 day before they should expire, so retrying in 30 minutes, then 2 hours, then 12 hours is fine. If even after that, the remote server does not work, there is little sense in retrying more often than once a day Also, uniqueness of the job should ensure that failed retries will not result in multiple retries for the same endpoint when the next resubscription cycle comes
2017-07-07Refactor JSON templates to be generated with ActiveModelSerializers instead ↵Eugen Rochko
of Rabl (#4090)
2017-06-25Bind web UI access tokens to sessions (#3940)Eugen Rochko
* Add overview of active sessions * Better display of browser/platform name * Improve how browser information is stored and displayed for sessions overview * Fix test * Fix #2347 - Bind web UI access token to session When you logout, session also destroys the access token, so it's no longer valid. If access token is destroyed some other way, the session is also destroyed, requiring a re-login. Fix #1681 - Add scheduler to remove revoked access tokens and grants * Fix test
2017-06-22Some minor change and spec for Account (#3813)Akihiko Odaki (@fn_aki@pawoo.net)
* Introduce domains method to Account relation Account had followers_domains method, which was excessively specific. Let relation of Account have domains method instead. * Move follow_mapping in Account to AccountInteractions * Introduce shared examples for AccountAvatar inclusion * Cover Account more
2017-06-19Rename FollowRemoteAccountService to ResolveRemoteAccountService (#3847)Eugen Rochko
Rename Activitypub to ActivityPub