about summary refs log tree commit diff
path: root/app/services/fetch_link_card_service.rb
AgeCommit message (Collapse)Author
2021-11-25Add trending links (#16917)Eugen Rochko
* Add trending links * Add overriding specific links trendability * Add link type to preview cards and only trend articles Change trends review notifications from being sent every 5 minutes to being sent every 2 hours Change threshold from 5 unique accounts to 15 unique accounts * Fix tests
2021-11-05Add support for structured data and more OpenGraph tags to link cards (#16938)Eugen Rochko
Save preview cards under their canonical URL Increase max redirects to follow from 2 to 3
2021-05-19Fix some RedisLocks auto-releasing too fast (#16276)Claire
* Fix Delete and Create-related locks expiring too fast Fixes #16238 By default, RedisLock expires after 10 seconds, which may not be enough to process statuses, especially when those have attached media files. This commit extends those 10 seconds to 15 minutes, which should be plenty enough to handle any status, while being short enough to not waste many sidekiq job retries in the exceedingly rare case in which a sidekiq process would crash when processing a `Create` or `Delete`. * Fix other RedisLock autorelease durations Fixes #15645 - things that only perform a few simple database queries (e.g. finding and saving a record) have been left unchanged, so they'll still use the default 10s duration - things that perform significantly more complex database queries have been changed to a 5 minutes timeout - things that perform multiple HTTP queries have been changed to a 15 minutes timeout
2021-03-04Fix URL scanning in note length validator and preview card fetching (#15827)Claire
* Add tests * Fix URL scanning in note length validator and preview card fetching
2021-03-02Update twitter-text from 1.14 to 3.1.0 and fix toot character counting (#15382)Claire
* Update twitter-text from 1.14 to 3.1.0 * Disable emoji parsing * Properly depend on twitter-text for url detection * Fix some URLs being wrongly detected client-side * Add test for server-side validation of non-autolinkable URLs * Fix server-side status length counting
2021-01-25Fix first return value of FetchLinkCardService.html method (#15630)Takeshi Umeda
2021-01-10Optimize map { ... }.compact calls (#15513)luigi
* Optimize map { ... }.compact using Enumerable#filter_map, supported since Ruby 2.7 * Add poyfill for Enumerable#filter_map
2020-09-01Bump rubocop from 0.86.0 to 0.88.0 (#14412)dependabot[bot]
* Bump rubocop from 0.86.0 to 0.88.0 Bumps [rubocop](https://github.com/rubocop-hq/rubocop) from 0.86.0 to 0.88.0. - [Release notes](https://github.com/rubocop-hq/rubocop/releases) - [Changelog](https://github.com/rubocop-hq/rubocop/blob/master/CHANGELOG.md) - [Commits](https://github.com/rubocop-hq/rubocop/compare/v0.86.0...v0.88.0) Signed-off-by: dependabot[bot] <support@github.com> * Fix for latest RuboCop Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yamagishi Kazutoshi <ykzts@desire.sh>
2020-07-07Change User-Agent of link preview fetching service to include "Bot" (#14248)Eugen Rochko
This forces Twitter to render OpenGraph tags in the response
2019-12-18Fix link crawler not specifying accepted content-type (#12646)ThibG
The link crawler expects HTML documents, so set the `Accept` header accordingly. Fixes #12618
2019-11-21Fix FetchLinkCardServices crashing on a tags without a target (#12159)ThibG
* Add test for links without targets * Fix FetchLinkCardServices crashing on a tags without a target
2019-11-17Add cache for OEmbed endpoints to avoid extra HTTP requests (#12403)Eugen Rochko
* add youtube oembed endpoint * add check for oembed endpoint * change unless for a more readable if * clear blank lines * endpoint via https * Fix string literal in condition * use cache for endpoints * use cache for endpoints * clean up and adding check * clean up and remove redundant return * add html check * add false to return * use double quotes * use double quotes * Clean up
2019-10-24microformat mentions can have an implicit property (#12189)nightpool
See the first example here: http://microformats.org/wiki/microformats2#hyperlinked_person
2019-10-01Remove HEAD request from fetching link previews (#12028)Eugen Rochko
It is not really necessary and we need to reduce requests
2019-09-28Fix preview card image not being re-fetched even if link is re-posted (#11981)Eugen Rochko
Fix #11956
2019-07-10Refactor fetching of remote resources (#11251)Eugen Rochko
2019-07-07Remove Atom feeds and old URLs in the form of `GET /:username/updates/:id` ↵Eugen Rochko
(#11247)
2019-04-21Treat meta[property] as a space-separated list (#10604)Daniel Aleksandersen
The @property attribute in HTML is a space-separated list of values. This change normalizes whitespace and finds the desired value in the list instead of requiring an exact single-value match. More details: https://www.ctrl.blog/entry/rdfa-socialmedia-metadata.html
2018-12-17 Ignore low-confidence CharlockHolmes guesses when parsing link cards (#9510)ThibG
* Add failing test for windows-1251 link cards * Ignore low-confidence CharlockHolmes guesses Fixes #9466 * Fix no method error when charlock holmes cannot detect charset
2018-11-10Check that twitter:player is valid before using it (#9254)ThibG
Fixes #9251
2018-10-30Fix Pleroma mentions being fetched as preview cards (#9158)ThibG
2018-10-28Include preview cards in status entity in REST API (#9120)Eugen Rochko
* Include preview cards in status entity in REST API * Display preview card in-stream * Improve in-stream display of preview cards
2018-10-25Do not fetch preview card for mentioned users (#6934)ThibG
2018-09-18Redesign forms, verify link ownership with rel="me" (#8703)Eugen Rochko
* Verify link ownership with rel="me" * Add explanation about verification to UI * Perform link verifications * Add click-to-copy widget for verification HTML * Redesign edit profile page * Redesign forms * Improve responsive design of settings pages * Restore landing page sign-up form * Fix typo * Support <link> tags, add spec * Fix links not being verified on first discovery and passive updates
2018-09-10Handle relative URLs when fetching OEmbed/OpenGraph cards (#8669)ThibG
2018-06-04Skip processing when HEAD method returns 501 (#7730)Yamagishi Kazutoshi
2018-05-16Raise Mastodon::RaceConditionError if Redis lock failed (#7511)Akihiko Odaki
An explicit error allows user agents to know the error and Sidekiq to retry.
2018-05-09Rescue Mastodon::LengthValidationError in FetchLinkCardService (#7424)Yamagishi Kazutoshi
2018-05-02Slightly reduce RAM usage (#7301)Eugen Rochko
* No need to re-require sidekiq plugins, they are required via Gemfile * Add derailed_benchmarks tool, no need to require TTY gems in Gemfile * Replace ruby-oembed with FetchOEmbedService Reduce startup by 45382 allocated objects * Remove preloaded JSON-LD in favour of caching HTTP responses Reduce boot RAM by about 6 MiB * Fix tests * Fix test suite by stubbing out JSON-LD contexts
2018-03-26Validate HTTP response length while receiving (#6891)Akihiko Odaki
to_s method of HTTP::Response keeps blocking while it receives the whole content, no matter how it is big. This means it may waste time to receive unacceptably large files. It may also consume memory and disk in the process. This solves the inefficency by checking response length while receiving.
2018-03-24Close http connection in perform method of Request class (#6889)Akihiko Odaki
HTTP connections must be explicitly closed in many cases, and letting perform method close connections makes its callers less redundant and prevent them from forgetting to close connections.
2018-02-15Fix #5173: Click card to embed external content (#6471)Eugen Rochko
2018-02-02Fix saving of oEmbed image (#6409)abcang
2017-12-13Fix redundant HTTP request in FetchLinkCardService (#6002)Eugen Rochko
2017-12-12Store preview image for embedded photo in preview cards (#5986)Akihiko Odaki
The preview image would be useful to embed in timeline.
2017-12-12Return false if object does not respond to url (#5988)Renato "Lond" Cerqueira
Avoid error when the service returns a mostly valid oembed, but has no url in it, causing a MethodError: undefined method `url' for #<OEmbed::Response::Photo:0x000056505def9620>
2017-12-09Ignore HEAD method if does not support (#5949)Yamagishi Kazutoshi
2017-12-07Add embed_url to preview cards (#5775)Akihiko Odaki
2017-11-20Fixed duplicating URL of photo type of oEmbed (#5763)abcang
2017-10-17Don't capture scheme-less URLs in the status (#5435)unarist
Specifically, this fixes status length calculation to be same as JS side. BTW, since this pattern used in not only preview card fetching, we should extract it (with twitter-regex?) and write tests I think.
2017-10-12Fix some failure cases on FetchLinkCardService (#5347)unarist
* If OEmbed response doesn't have a required property `type`, ignore it. e.g. `NoMethodError: undefined method 'type' for ...` * If we failed to detect encoding, fallback to default behavior of Nokogiri. e.g. `KeyError: key not found: :encoding`
2017-10-06Improve error handling on LinkCrawlWorker (#5250)unarist
* Improve error handling on LinkCrawlWorker * Ignore TimeoutError and InvalidURIError too * Record errors to debug log * Enable dead job queue on LinkCrawlWorker Since most of acceptable errors were already ignored, only our side issue should go to dead job queue. * Ignore all http gem errors
2017-09-14Enable to recognize most kinds of characters as URL paths (#4941)ふぁぼ原
2017-09-14Support OpenGraph video embeds (#4897)Eugen Rochko
* Support OpenGraph video embeds It's not really OpenGraph, it's twitter:player property, but it's not OEmbed so that fits. For example, this allows Twitch clips to be displayed as embeds. Also, fixes glitch-soc/mastodon#135 * Fix invalid OpenGraph cards being saved through attaching and revisit URLs after 14 days
2017-09-01Make PreviewCard records reuseable between statuses (#4642)Eugen Rochko
* Make PreviewCard records reuseable between statuses **Warning!** Migration truncates preview_cards tablec * Allow a wider thumbnail for link preview, display it in horizontal layout (#4648) * Delete preview cards files before truncating * Rename old table instead of truncating it * Add mastodon:maintenance:remove_deprecated_preview_cards * Ignore deprecated_preview_cards in schema definition * Fix null behaviour
2017-07-14HTTP signatures (#4146)Eugen Rochko
* Add Request class with HTTP signature generator Spec: https://tools.ietf.org/html/draft-cavage-http-signatures-06 * Add HTTP signature verification concern * Add test for SignatureVerification concern * Add basic test for Request class * Make PuSH subscribe/unsubscribe requests use new Request class Accidentally fix lease_seconds not being set and sent properly, and change the new minimum subscription duration to 1 day * Make all PuSH workers use new Request class * Make Salmon sender use new Request class * Make FetchLinkService use new Request class * Make FetchAtomService use the new Request class * Make Remotable use the new Request class * Make ResolveRemoteAccountService use the new Request class * Add more tests * Allow +-30 seconds window for signed request to remain valid * Disable time window validation for signed requests, restore 7 days as PuSH subscription duration (which was previous default due to a bug)
2017-07-08Use charlock_holmes instead of nkf at FetchLinkCardService (#4080)nullkal
* Specs for language detection * Use CharlockHolmes instead of NKF * Correct mistakes * Correct style * Set hint_enc instead of falling back and strip_tags * Improve specs * Add dependencies
2017-07-05Fix Nokogiri::HTML at FetchLinkCardService (#4072)abcang
2017-07-03Rescue exceptions caused by FetchLinkCardService (#4045)abcang
2017-06-11Fix an error when TagManager.local_url? is called with a bad URI (#3701)ThibG
TagManager.local_url? was sometimes called with an URI with a nil host, leading to a crash in TagManager.local_url?. This fixes moves the already-existing uri.host.blank? check in front to avoid this case.
generated by cgit-pink (git 2.37.1) at 2024-11-07 13:35:22 +0000