Repository navigation
Add Welsh Language (cy) - #14044
Add Welsh Language (cy)#14044mgrbyte wants to merge 3 commits into
Conversation
- Welsh language class and the cy_tokenizer fixture
- punctuation.py: f'/d' prefixes, clitic suffixes ('r 'n 'i 'm 'ch 'th 'u 'w)
with straight and curly apostrophes; an explicit infix list that keeps
hyphens within words and splits them between digits
- tokenizer_exceptions.py: e.e., h.y., a.y.b., Dr., Mr., Mrs. with NORM, and
the clitics standing alone
- Tests in spacy/tests/lang/cy: prefixes, suffixes, hyphens, wrapping
punctuation, abbreviations, sentences
- Ref: explosion#1
- lex_attrs.py: like_num with the number words and their mutations - stop_words.py: the Unit's list grouped by part of speech, with eu and un - examples.py and the cy entry in website/meta/languages.json - WelshDefaults wires them in - Tests: like_num (digits, fractions, mutated number words, non-numbers), stop words (the eighteen most frequent corpus words), and is_stop and like_num through the tokenizer - Ref: explosion#1
chrikrah
left a comment
There was a problem hiding this comment.
@mgrbyte this looks ready to me. One consistency fix I'd make first, and one question.
non-blocking: punctuation.py splits ’r and ’n like 'r and 'n, but stop_words.py lists only the straight forms. The loop at spacy/lang/en/stop_words.py:71 adds a ‘ and a ’ copy of each contraction.
$ python -I probe_cy.py # spacy 3.8.16 wheel, spacy/lang/cy and spacy/tests from 66aab71 copied over it
"Mae'r gath yn cysgu ar y mat" -> ("'r", is_stop=True)
'Mae’r gath yn cysgu ar y mat' -> ('’r', is_stop=False)
'Cafodd ei ysgrifennu gan Dafydd' -> ('gan', is_stop=True, like_num=True)
$ python -m pytest spacy/tests/lang/cy -q
94 passed # 66aab71
94 passed # 66aab71 plus the curly copies and gan dropped; '’r' then reads is_stop=True
non-blocking: lex_attrs.py puts gan in _num_words as the mutated can. Your CORPUS_FREQUENT test lists gan as a top word, and there it is mostly the preposition. Do you want like_num on every gan?
- The tokenizer splits ’r and ’n like 'r and 'n; both spellings are now stop words, so is_stop no longer depends on which apostrophe was typed - Tests: all four apostrophe variants are stop words, and is_stop through the tokenizer is the same for straight and curly apostrophes
|
Thanks for the review. The curly-apostrophe stop words are fixed in the latest commit. On |
Description
This PR adds Welsh (
cy) tospacy/lang: tokenizer rules, tokenizer exceptions, stop words,lexical attributes and example sentences, with tests. It replaces nothing; Welsh has had no
language data in spaCy before. The work comes from the Language Technologies Unit at Bangor
University, Wales (UK) which develops and maintains Welsh language technologies.
What is included:
spacy/lang/cy/__init__.py: theWelshlanguage class andWelshDefaults.punctuation.py: Welsh clitics are written with an apostrophe and attach to the preceding word(
'r,'n,'i,'m,'ch,'th,'u,'w) or, forf'andd', to the following one.They are split into their own tokens, with straight and curly apostrophes, matching the
established tokenisation seen in software such as the Cysill Welsh-language spelling and grammar checker (
i'r→i+'r,Mae'r→Mae+'r,f'annwylyd→f'+annwylyd). Welsh hyphenates within words to mark stress and to separatedigraphs from letter pairs (
cyd-destunhas two Ds,cuddioone DD as DD is a letter in its own right in Welsh), so the base infix rulethat splits on a hyphen between letters is left out; hyphens between digits still split.
tokenizer_exceptions.py: common abbreviations withNORM(e.e.er enghraifft,h.y.hynny yw,
a.y.b.ac yn y blaen,Dr.,Mr.,Mrs.), and the clitics as standalone tokensso the quote prefix rule does not strip their apostrophe.
lex_attrs.py:like_numcovering digits, fractions and the Welsh number words including theirsoft, nasal and aspirate mutations (
dau/ddau/nau,cant/gant/chant, …).stop_words.py: a stop-word list grouped by part of speech based on the type of word found in the equivalent English file but expanded to allow for conjugated verbal forms and mutated forms.examples.py, and thecyentry inwebsite/meta/languages.json.spacy/tests/lang/cy/: 94 tests (prefixes, suffixes, hyphens, wrapping punctuation,abbreviations, sentences,
like_num, stop words), plus thecy_tokenizerfixture inspacy/tests/conftest.py.Not included, deliberately: a lemmatizer. The Unit's lemma tables (CC0) are being prepared for
spacy-lookups-data; the lemmatizer and its tables will follow as a separate PR pair so this onestays self-contained. There are no
syntax_iteratorsyet either, as there is currently no dependency parserto drive them (note that the existing Welsh UD-CCG follows a different but valid POS tagging implementation which unfortunately does not enable differentiation between nominal and verbal verbnouns using the POS tags).
We are also preparing to make available our POS tagging and NER models for Welsh.
Testing:
pytest spacy/tests/lang/cy(94 passed) and the full suite(
pytest spacy: 3711 passed, 1229 skipped) on Python 3.13, on macOS and Linux, with-W error;ruff format --check,ruff check --select Iandmypy spacyare clean.Types of change
New feature: language support for Welsh (
cy).Checklist