Fix language filter codes (#4841)

* Fix language filter codes

CLD3 returns BCP-47 language identifier, filter settings expect
identifiers in the ISO 639-1 format. Convert between formats,
and exclude duplicate languages from filter choices (zh-CN->zh)

* Fix zh name

1 files changed, 10 insertions, 1 deletions

diff --git a/app/lib/language_detector.rb b/app/lib/language_detector.rb
index cc7509fdc..1d9932b52 100644
--- a/app/lib/language_detector.rb
+++ b/app/lib/language_detector.rb
@@ -20,7 +20,16 @@ class LanguageDetector
   private
 
   def detected_language_code
-    result.language.to_sym if detected_language_reliable?
+    iso6391(result.language).to_sym if detected_language_reliable?
+  end
+
+  def iso6391(bcp47)
+    iso639 = bcp47.split('-').first
+
+    # CLD3 returns grandfathered language code for Hebrew
+    return 'he' if iso639 == 'iw'
+
+    ISO_639.find(iso639).alpha2
   end
 
   def result