about summary refs log tree commit diff
path: root/app/workers/scheduler
AgeCommit message (Collapse)Author
2021-08-09Merge branch 'main' into glitch-soc/merge-upstreamClaire
Conflicts: - `Gemfile.lock`: Not a real conflict, upstream-updated dependency (redis) textually too close to glitch-soc-only dependecy. Updated redis gem like upstream did.
2021-08-09Add feature to automatically delete old toots (#16529)Claire
* Add account statuses cleanup policy model * Record last inspected toot to delete to speed up successive calls to statuses_to_delete * Add service to cleanup a given account's statuses within a budget * Add worker to go through account policies and delete old toots * Fix last inspected status id logic All existing statuses older or equal to last inspected status id must be kept by the current policy. This is an invariant that must be kept so that resuming deletion from the last inspected status remains sound. * Add tests * Refactor scheduler and add tests * Add user interface * Add support for discriminating based on boosts/favs * Add UI support for min_reblogs and min_favs, rework UI * Address first round of review comments * Replace Snowflake#id_at_start with with_random parameter * Add tests * Add tests for StatusesCleanupController * Rework settings page * Adjust load-avoiding mechanisms * Please CodeClimate
2021-06-23Merge branch 'main' into glitch-soc/merge-upstreamClaire
2021-06-21Add authentication history (#16408)Eugen Rochko
2021-05-07Merge branch 'main' into glitch-soc/merge-upstreamClaire
- `app/views/statuses/_simple_status.html.haml`: Small markup change in glitch-soc, on a line that has been modified by upstream. Ported upstream changes.
2021-05-05Improve performance of follow recommendation scheduler (#16159)Claire
Express follow_recommendations in terms of account_summaries rather than accounts, integrate filters that are unconditionally used, and materialize the resulting view. This should result in the bulk of the computation being performed only once instead of **once per recommendation language**.
2021-04-20Merge branch 'main' into glitch-soc/merge-upstreamClaire
Conflicts: - `README.md`: Upstream updated copyright year, we don't mention it so kept our version. - `app/controllers/admin/dashboard_controller.rb`: Not really a conflict, upstream change (removing the spam checker) too close to glitch-soc changes. Ported upstream changes. - `app/models/form/admin_settings.rb`: Same. - `app/services/remove_status_service.rb`: Same. - `app/views/admin/settings/edit.html.haml`: Same. - `config/settings.yml`: Same. - `config/environments/production.rb`: Not a real conflict, upstream added a default HTTP header, but we have extra headers in glitch-soc. Added the header.
2021-04-12Add cold-start follow recommendations (#15945)Eugen Rochko
2021-03-15Merge branch 'main' into glitch-soc/merge-upstreamClaire
2021-03-15Fix configuration for sidekiq-unique-jobs after 7.x upgrade (#15908)Eugen Rochko
Remove locks from scheduled jobs
2020-12-23Merge branch 'master' into glitch-soc/merge-upstreamClaire
Conflicts: - `app/lib/feed_manager.rb`: Not a real conflict, glitch-soc-only DM-related method too close to changed upstream stuff. Ported upstream changes. - `app/services/batched_remove_status_service.rb`: Additional logic in glitch-soc to clear DMs from timelines. Ported upstream changes and fixed the DM TL clearing logic. - `app/workers/scheduler/feed_cleanup_scheduler.rb`: Additional code in glitch-soc to clear DM timelines. Ported upstream changes.
2020-12-22Improve account deletion performances further (#15407)ThibG
* Delete status records by batches of 50 * Do not precompute values that are only used once * Do not generate redis events for removal of public toots older than two weeks * Filter reported toots a priori for polls and status deletion * Do not process reblogs when cleaning up public timelines As in Mastodon proper, reblogs don't appear in public TLs * Clean the deleted account's own feed in one go * Refactor Account#clean_feed_manager and List#clean_feed_manager * Delete instead of destroy a few more associations * Fix preloading Co-authored-by: Claire <claire.github-309c@sitedethib.com>
2020-12-15Merge branch 'master' into glitch-soc/merge-upstreamClaire
Conflicts: - `app/models/form/admin_settings.rb`: New setting added upstream. Ported it. - `app/views/statuses/_simple_status.html.haml`: Upstream removed RTL classes. Did the same. - `config/settings.yml`: New setting added upstream. Ported it.
2020-12-14Fix performance on instances list in admin UI (#15282)Eugen Rochko
- Reduce duplicate queries - Remove n+1 queries - Add accounts count to detailed view - Add separate action log entry for updating existing domain blocks
2020-10-21Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - `.github/dependabot.yml`: Updated upstream, we deleted it to not be flooded by Depandabot. Kept deleted. - `Gemfile.lock`: Puma updated on both sides, went for the most recent version. - `app/controllers/api/v1/mutes_controller.rb`: Upstream updated the serializer to support timed mutes, while glitch-soc added a custom API ages ago to get information that is already available elsewhere. Dropped the glitch-soc-specific API, went with upstream changes. - `app/javascript/core/admin.js`: Conflict due to changing how assets are loaded. Went with upstream. - `app/javascript/packs/public.js`: Conflict due to changing how assets are loaded. Went with upstream. - `app/models/mute.rb`: 🤷 - `app/models/user.rb`: New user setting added upstream while we have glitch-soc-specific user settings. Added upstream's user setting. - `config/settings.yml`: Upstream added a new user setting close to a user setting we had changed the defaults for. Added the new upstream setting. - `package.json`: Upstream dependency updated “too close” to a glitch-soc-specific dependency. No real conflict. Updated the dependency.
2020-10-12Add IP-based rules (#14963)Eugen Rochko
2020-09-28Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - `Gemfile.lock`: Not a real conflict, upstream updated dependencies that were too close to glitch-soc-only ones in the file. - `app/controllers/oauth/authorized_applications_controller.rb`: Upstream changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc's theming system. Ported upstream changes. - `app/controllers/settings/base_controller.rb`: Upstream refactored and changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc's theming system. Ported upstream changes. - `app/controllers/settings/sessions_controller.rb`: Upstream refactored and changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc's theming system. Ported upstream changes. - `app/models/user.rb`: Upstream refactored and changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc not preventing moved accounts from logging in. Ported upstream changes while keeping the ability for moved accounts to log in. - `app/policies/status_policy.rb`: Upstream refactored and changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc's local-only toots. Ported upstream changes. - `app/serializers/rest/account_serializer.rb`: Upstream refactored and changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc's ability to hide followers count. Ported upstream changes. - `app/services/process_mentions_service.rb`: Upstream refactored and changed the logic surrounding suspended accounts. Minor conflict due to glitch-soc's local-only toots. Ported upstream changes. - `package.json`: Not a real conflict, upstream updated dependencies that were too close to glitch-soc-only ones in the file.
2020-09-15Change account suspensions to be reversible by default (#14726)Eugen Rochko
2020-06-09Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - `app/controllers/activitypub/collections_controller.rb`: Conflict due to glitch-soc having to take care of local-only pinned toots in that controller. Took upstream's changes and restored the local-only special handling. - `app/controllers/auth/sessions_controller.rb`: Minor conflicts due to the theming system, applied upstream changes, adapted the following two files for glitch-soc's theming system: - `app/controllers/concerns/sign_in_token_authentication_concern.rb` - `app/controllers/concerns/two_factor_authentication_concern.rb` - `app/services/backup_service.rb`: Minor conflict due to glitch-soc having to handle local-only toots specially. Applied upstream changes and restored the local-only special handling. - `app/views/admin/custom_emojis/index.html.haml`: Minor conflict due to the theming system. - `package.json`: Upstream dependency updated, too close to a glitch-soc-only dependency in the file. - `yarn.lock`: Upstream dependency updated, too close to a glitch-soc-only dependency in the file.
2020-06-02Add E2EE API (#13820)Eugen Rochko
2020-04-02Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - `app/javascript/mastodon/features/compose/components/poll_form.js`: Upstream bumped poll option character limit, but we already had a higher one, kept ours. - `app/validators/poll_validator.rb`: Upstream bumped poll option character limit, but we already had a higher one, kept ours. - `config/initializers/content_security_policy.rb`: Upstream added a rule, the way we compute ours is different, but that added rule has been ported. - `package.json`: No real conflict, dependency update. Performed the same update. - `yarn.lock`: No real conflict, dependency update. Performed the same update.
2020-03-31Fix background jobs not using locks like they are supposed to (#13361)Eugen Rochko
Also: - Fix locks not being removed when jobs go to the dead job queue - Add UI for managing locks to the Sidekiq dashboard - Remove unused Sidekiq workers Fix #13349
2020-01-27Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - `app/serializers/rest/account_serializer.rb`: Upstream added code too close to glitch-soc-specific followers-hiding code. Ported upstream changes.
2020-01-26Fix expired announcements being re-published (#12964)Eugen Rochko
2020-01-24Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - `app/controllers/statuses_controller.rb`: Minor conflict due to theming system
2020-01-23Add announcements (#12662)Eugen Rochko
* Add announcements Fix #11006 * Add reactions to announcements * Add admin UI for announcements * Add unit tests * Fix issues - Add `with_dismissed` param to announcements API - Fix end date not being formatted when time range is given - Fix announcement delete causing reactions to send streaming updates - Fix announcements container growing too wide and mascot too small - Fix `all_day` being settable when no time range is given - Change text "Update" to "Announcement" * Fix scheduler unpublishing announcements before they are due * Fix filter params not being passed to announcements filter
2019-09-18Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - app/controllers/auth/sessions_controller.rb Minor conflict due to glitch-soc's theming code
2019-09-17Improve IP cleanup query (#11871)abcang
2019-09-05Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
2019-09-02Change trending hashtags to not disappear instantly after midnight (#11712)Eugen Rochko
2019-07-28Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
2019-07-28Add `tootctl preview_cards remove` (#11320)mayaeh
* Add `tootctl preview_cards remove` * fix code style * Remove `Scheduler::PreviewCardsCleanupScheduler` file * fix code style again Add exclude case where image_file_name is blank * Added a function to output confirmation if the specified number of days is less than 2 weeks
2019-07-23Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - app/controllers/application_controller.rb - app/controllers/auth/confirmations_controller.rb - app/controllers/auth/sessions_controller.rb - app/controllers/settings/deletes_controller.rb - app/controllers/settings/two_factor_authentication/recovery_codes_controller.rb
2019-07-21Remove WebSub subscriptions (#11303)Eugen Rochko
2019-07-19Merge branch 'master' into glitch-soc/merge-upstreamThibaut Girka
Conflicts: - Gemfile.lock - app/controllers/accounts_controller.rb - app/controllers/admin/dashboard_controller.rb - app/controllers/follower_accounts_controller.rb - app/controllers/following_accounts_controller.rb - app/controllers/remote_follow_controller.rb - app/controllers/stream_entries_controller.rb - app/controllers/tags_controller.rb - app/javascript/packs/public.js - app/lib/sanitize_config.rb - app/models/account.rb - app/models/form/admin_settings.rb - app/models/media_attachment.rb - app/models/stream_entry.rb - app/models/user.rb - app/serializers/initial_state_serializer.rb - app/services/batched_remove_status_service.rb - app/services/post_status_service.rb - app/services/process_mentions_service.rb - app/services/reblog_service.rb - app/services/remove_status_service.rb - app/views/admin/settings/edit.html.haml - config/locales/simple_form.pl.yml - config/settings.yml - docker-compose.yml
2019-07-15Add periodic removal of older thumbnails for preview cards (#11304)Eugen Rochko
2019-07-06Remove Salmon and PubSubHubbub (#11205)Eugen Rochko
* Remove Salmon and PubSubHubbub endpoints * Add error when trying to follow OStatus accounts * Fix new accounts not being created in ResolveAccountService
2019-06-27Use a redis-cached feed for the DM timelineThibaut Girka
2019-02-02Create Redisable#redis (#9633)ysksn
* Create Redisable * Use #redis instead of Redis.current
2019-01-06Fix PublishScheduledStatusWorker being queued with no arguments (#9724)Eugen Rochko
2019-01-05Add scheduled statuses (#9706)Eugen Rochko
Fix #340
2019-01-02Improve e-mail digest (#9689)Eugen Rochko
- Reduce time-to-digest from 20 to 7 days - Fetch mentions starting from +1 day since last login - Fix case when last login is more recent than last e-mail - Do not render all mentions, only 40, but show number in subject - Do not send digest to moved accounts - Do send digest to silenced accounts
2018-10-09Track historical space stats in PgHero to determine PostgreSQL growth (#8906)Eugen Rochko
2018-08-25Disable retries on all scheduled tasks (#8413)Eugen Rochko
2018-08-21Get rid of all batch order warnings (#8334)Eugen Rochko
2018-08-19Keep scheduler jobs unique until they're done (#8287)Eugen Rochko
2018-05-02Slightly reduce RAM usage (#7301)Eugen Rochko
* No need to re-require sidekiq plugins, they are required via Gemfile * Add derailed_benchmarks tool, no need to require TTY gems in Gemfile * Replace ruby-oembed with FetchOEmbedService Reduce startup by 45382 allocated objects * Remove preloaded JSON-LD in favour of caching HTTP responses Reduce boot RAM by about 6 MiB * Fix tests * Fix test suite by stubbing out JSON-LD contexts
2018-04-04Adjust privacy policy (#6666)Eugen Rochko
* Adjust privacy policy to be more specific to Mastodon Fix #6613 * Change data retention of IP addresses from 5 years to 1 year * Add even more information * Remove all (now invalid) translations of the privacy policy * Add information about archive takeout, remove pointless consent section * Emphasis on DM privacy * Improve wording * Add line about data use for moderation purposes
2018-02-21Account archive download (#6460)Eugen Rochko
* Fix #201: Account archive download * Export actor and private key in the archive * Optimize BackupService - Add conversation to cached associations of status, because somehow it was forgotten and is source of N+1 queries - Explicitly call GC between batches of records being fetched (Model class allocations are the worst offender) - Stream media files into the tar in 1MB chunks (Do not allocate media file (up to 8MB) as string into memory) - Use #bytesize instead of #size to calculate file size for JSON (Fix FileOverflow error) - Segment media into subfolders by status ID because apparently GIF-to-MP4 media are all named "media.mp4" for some reason * Keep uniquely generated filename in Paperclip::GifTranscoder * Ensure dumped files do not overwrite each other by maintaing directory partitions * Give tar archives a good name * Add scheduler to remove week-old backups * Fix code style issue
2018-01-15Move e-mail digest task to sidekiq, reduce workload, improve hint (#6252)Eugen Rochko