From 4a64181461cb02599da98166da4b527adbb705ad Mon Sep 17 00:00:00 2001 From: David Yip Date: Sun, 15 Oct 2017 19:49:22 -0500 Subject: Allow keywords to match either substrings or whole words. Word-boundary matching only works as intended in English and languages that use similar word-breaking characters; it doesn't work so well in (say) Japanese, Chinese, or Thai. It's unacceptable to have a feature that doesn't work as intended for some languages. (Moreso especially considering that it's likely that the largest contingent on the Mastodon bit of the fediverse speaks Japanese.) There are rules specified in Unicode TR29[1] for word-breaking across all languages supported by Unicode, but the rules deliberately do not cover all cases. In fact, TR29 states For example, reliable detection of word boundaries in languages such as Thai, Lao, Chinese, or Japanese requires the use of dictionary lookup, analogous to English hyphenation. So we aren't going to be able to make word detection work with regexes within Mastodon (or glitchsoc). However, for a first pass (even if it's kind of punting) we can allow the user to choose whether they want word or substring detection and warn about the limitations of this implementation in, say, docs. [1]: https://unicode.org/reports/tr29/ https://web.archive.org/web/20171001005125/https://unicode.org/reports/tr29/ --- spec/models/keyword_mute_spec.rb | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) (limited to 'spec/models') diff --git a/spec/models/keyword_mute_spec.rb b/spec/models/keyword_mute_spec.rb index de5d32bb4..c74505188 100644 --- a/spec/models/keyword_mute_spec.rb +++ b/spec/models/keyword_mute_spec.rb @@ -30,10 +30,16 @@ RSpec.describe KeywordMute, type: :model do expect(matcher =~ 'This is a hot take').to be_falsy end - it 'does not match substrings matching keywords' do - KeywordMute.create!(account: alice, keyword: 'take') + it 'considers word boundaries when matching' do + KeywordMute.create!(account: alice, keyword: 'bob', whole_word: true) + + expect(matcher =~ 'bobcats').to be_falsy + end + + it 'matches substrings if whole_word is false' do + KeywordMute.create!(account: alice, keyword: 'take', whole_word: false) - expect(matcher =~ 'This is a shiitake mushroom').to be_falsy + expect(matcher =~ 'This is a shiitake mushroom').to be_truthy end it 'matches keywords at the beginning of the text' do -- cgit