Skip to content

Add Welsh Language (cy) - #14044

Open
mgrbyte wants to merge 3 commits into
explosion:masterfrom
techiaith:welsh
Open

mgrbyte wants to merge 3 commits into
explosion:masterfrom
techiaith:welsh

Conversation

@mgrbyte

@mgrbyte mgrbyte commented Oct 5, 2026

Copy link
Copy Markdown

Description

This PR adds Welsh (cy) to spacy/lang: tokenizer rules, tokenizer exceptions, stop words,
lexical attributes and example sentences, with tests. It replaces nothing; Welsh has had no
language data in spaCy before. The work comes from the Language Technologies Unit at Bangor
University, Wales (UK) which develops and maintains Welsh language technologies.

What is included:

  • spacy/lang/cy/__init__.py: the Welsh language class and WelshDefaults.
  • punctuation.py: Welsh clitics are written with an apostrophe and attach to the preceding word
    ('r, 'n, 'i, 'm, 'ch, 'th, 'u, 'w) or, for f' and d', to the following one.
    They are split into their own tokens, with straight and curly apostrophes, matching the
    established tokenisation seen in software such as the Cysill Welsh-language spelling and grammar checker (i'r → i + 'r, Mae'r → Mae + 'r,
    f'annwylyd → f' + annwylyd). Welsh hyphenates within words to mark stress and to separate
    digraphs from letter pairs (cyd-destun has two Ds, cuddio one DD as DD is a letter in its own right in Welsh), so the base infix rule
    that splits on a hyphen between letters is left out; hyphens between digits still split.
  • tokenizer_exceptions.py: common abbreviations with NORM (e.e. er enghraifft, h.y.
    hynny yw, a.y.b. ac yn y blaen, Dr., Mr., Mrs.), and the clitics as standalone tokens
    so the quote prefix rule does not strip their apostrophe.
  • lex_attrs.py: like_num covering digits, fractions and the Welsh number words including their
    soft, nasal and aspirate mutations (dau/ddau/nau, cant/gant/chant, …).
  • stop_words.py: a stop-word list grouped by part of speech based on the type of word found in the equivalent English file but expanded to allow for conjugated verbal forms and mutated forms.
  • examples.py, and the cy entry in website/meta/languages.json.
  • spacy/tests/lang/cy/: 94 tests (prefixes, suffixes, hyphens, wrapping punctuation,
    abbreviations, sentences, like_num, stop words), plus the cy_tokenizer fixture in
    spacy/tests/conftest.py.

Not included, deliberately: a lemmatizer. The Unit's lemma tables (CC0) are being prepared for
spacy-lookups-data; the lemmatizer and its tables will follow as a separate PR pair so this one
stays self-contained. There are no syntax_iterators yet either, as there is currently no dependency parser
to drive them (note that the existing Welsh UD-CCG follows a different but valid POS tagging implementation which unfortunately does not enable differentiation between nominal and verbal verbnouns using the POS tags).

We are also preparing to make available our POS tagging and NER models for Welsh.

Testing: pytest spacy/tests/lang/cy (94 passed) and the full suite
(pytest spacy: 3711 passed, 1229 skipped) on Python 3.13, on macOS and Linux, with
-W error; ruff format --check, ruff check --select I and mypy spacy are clean.

Types of change

New feature: language support for Welsh (cy).

Checklist

  • I confirm that I have the right to submit this contribution under the project's MIT license.
  • I ran the tests, and all new and existing tests passed.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

- Welsh language class and the cy_tokenizer fixture
- punctuation.py: f'/d' prefixes, clitic suffixes ('r 'n 'i 'm 'ch 'th 'u 'w)
  with straight and curly apostrophes; an explicit infix list that keeps
  hyphens within words and splits them between digits
- tokenizer_exceptions.py: e.e., h.y., a.y.b., Dr., Mr., Mrs. with NORM, and
  the clitics standing alone
- Tests in spacy/tests/lang/cy: prefixes, suffixes, hyphens, wrapping
  punctuation, abbreviations, sentences
- Ref: explosion#1
- lex_attrs.py: like_num with the number words and their mutations
- stop_words.py: the Unit's list grouped by part of speech, with eu and un
- examples.py and the cy entry in website/meta/languages.json
- WelshDefaults wires them in
- Tests: like_num (digits, fractions, mutated number words, non-numbers),
  stop words (the eighteen most frequent corpus words), and is_stop and
  like_num through the tokenizer
- Ref: explosion#1

@chrikrah chrikrah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgrbyte this looks ready to me. One consistency fix I'd make first, and one question.

non-blocking: punctuation.py splits ’r and ’n like 'r and 'n, but stop_words.py lists only the straight forms. The loop at spacy/lang/en/stop_words.py:71 adds a ‘ and a ’ copy of each contraction.

$ python -I probe_cy.py   # spacy 3.8.16 wheel, spacy/lang/cy and spacy/tests from 66aab71 copied over it
"Mae'r gath yn cysgu ar y mat"    -> ("'r", is_stop=True)
'Mae’r gath yn cysgu ar y mat'    -> ('’r', is_stop=False)
'Cafodd ei ysgrifennu gan Dafydd' -> ('gan', is_stop=True, like_num=True)
$ python -m pytest spacy/tests/lang/cy -q
94 passed                 # 66aab71
94 passed                 # 66aab71 plus the curly copies and gan dropped; '’r' then reads is_stop=True

non-blocking: lex_attrs.py puts gan in _num_words as the mutated can. Your CORPUS_FREQUENT test lists gan as a top word, and there it is mostly the preposition. Do you want like_num on every gan?

- The tokenizer splits ’r and ’n like 'r and 'n; both spellings are now stop
  words, so is_stop no longer depends on which apostrophe was typed
- Tests: all four apostrophe variants are stop words, and is_stop through the
  tokenizer is the same for straight and curly apostrophes
@mgrbyte

mgrbyte commented Oct 15, 2026

Copy link
Copy Markdown
Author

Thanks for the review. The curly-apostrophe stop words are fixed in the latest commit.

On gan: we've kept it, along with the other mutated numerals that share a spelling with
another word (deg, un). like_num is a lexical attribute computed without context, so a
static list can't separate the numeral from its homograph; that is the tagger's job (NUM versus
ADP). This matches how other languages handle their most frequent case, the indefinite article:
un/une in French, een in Dutch, en in Swedish and Danish, bir in Turkish and ένα in
Greek are all like_num=True. Dropping gan would also lose real numerals such as dau gan.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants