pyinstaller hidden-imports for spacy deployment (windows)

[DarkSoliditi]

Hey, I'm having a difficult time finding information for making pyinstaller deploy spacy dependencies for windows. When I run
pyinstaller main.py --noconsole and run that everything in my app works except for the spacy functionality. I don't get an error or anything it just hits the spacy lines of code and silently fails.

I've tried to make a hidden-imports .py file. (code attached) but when I run pyinstaller main.py --additional-hooks-dir=. myhooks.py
it says pyinstaller: error: unrecognized arguments: myhooks.py

I'm not sure what I'm doing wrong or if I'm on the right path to get this working or not. I'm not looking for anyone to do the work for me but some direction would be appreciated.

Thanks in advance.

# HOOK FILE FOR SPACY
from PyInstaller.utils.hooks import collect_all

# ----------------------------- SPACY -----------------------------
data = collect_all('spacy')

datas = data[0]
binaries = data[1]
hiddenimports = data[2]

# ----------------------------- THINC -----------------------------
data = collect_all('thinc')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- CYMEM -----------------------------
data = collect_all('cymem')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- PRESHED -----------------------------
data = collect_all('preshed')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- BLIS -----------------------------

data = collect_all('blis')

datas += data[0]
binaries += data[1]
hiddenimports += data[2]

# ----------------------------- OTHER ----------------------------

hiddenimports += ['srsly.msgpack.util']


from PyInstaller.utils.hooks import collect_data_files
datas = collect_data_files("en_core_web_trf")

[polm]

Sorry you're having trouble with this, but I don't think any of us on the dev team are very familiar with PyInstaller. There are a few old issues/discussions about it, and many answers on Stack Overflow, but I think solving the problem has more to do with how PyInstaller works than how spaCy does. You might be better off asking in their community if there is one.

One thing that seems to come up in Stack Overflow issue a lot is that spaCy works but finding models doesn't, so maybe try using spacy.blank and seeing if that works (even if your app isn't usable that way). That would help you narrow down the issue at least.

Sorry I can't be of more help.

[DarkSoliditi]

I'll add that to my notes and I'm sorry I actually meant to post this on the pyinstaller github discussion board but I had so many tabs open I must have confused them. Thanks for the reply. I will get it working and post my solution here.

[DarkSoliditi]

Sorry you're having trouble with this, but I don't think any of us on the dev team are very familiar with PyInstaller. There are a few old issues/discussions about it, and many answers on Stack Overflow, but I think solving the problem has more to do with how PyInstaller works than how spaCy does. You might be better off asking in their community if there is one.

One thing that seems to come up in Stack Overflow issue a lot is that spaCy works but finding models doesn't, so maybe try using spacy.blank and seeing if that works (even if your app isn't usable that way). That would help you narrow down the issue at least.

Sorry I can't be of more help.

Hey I've been working with a pyinstaller dev and I've made some progress but I got this error in the Exception Callback and it's a spacy error code. It's still going to be a pyinstaller workaround because it works perfect in the development environment but the remedy on this discussion board is to install spacy-transformers but I can't since it's the deployment process making the problem. If this wasn't a deployment issue with pyinstaller what would normally causes this error code?

Exception Callback attached:
Exception in Tkinter callback
Traceback (most recent call last):
File "tkinter_init_.py", line 1892, in call
File "main.py", line 78, in tesseract
File "spacy_init_.py", line 51, in load
File "spacy\util.py", line 321, in load_model
File "spacy\util.py", line 354, in load_model_from_package
File "en_core_web_trf_init_.py", line 10, in load
File "spacy\util.py", line 514, in load_model_from_init_py
File "spacy\util.py", line 389, in load_model_from_path
File "spacy\util.py", line 426, in load_model_from_config
File "spacy\language.py", line 1715, in from_config
File "spacy\language.py", line 777, in add_pipe
File "spacy\language.py", line 641, in create_pipe
ValueError: [E002] Can't find factory for 'transformer' for language English (en
). This usually happens when spaCy calls nlp.create_pipe with a custom compone
nt name that's not registered on the current language class. If you're using a T
ransformer, make sure to install 'spacy-transformers'. If you're using a custom
component, make sure you've added the decorator @Language.component (for funct
ion components) or @Language.factory (for class components).

Available factories: attribute_ruler, tok2vec, merge_noun_chunks, merge_entities
, merge_subtokens, token_splitter, parser, beam_parser, entity_linker, ner, beam
_ner, entity_ruler, lemmatizer, tagger, morphologizer, senter, sentencizer, text
cat, spancat, textcat_multilabel, en.lemmatizer

[DarkSoliditi]

I've got a solution, Rokm on the pyinstaller discussion board found an implementation that works.

Here is the terminal command to deploy an app
pyinstaller --clean --noconfirm your_project.py --collect-submodules en_core_web_trf --collect-data en_core_web_trf --copy-metadata en_core_web_trf --collect-submodules spacy_transformers --copy-metadata spacy_transformers --copy-metadata tqdm --copy-metadata regex --copy-metadata sacremoses --copy-metadata requests --copy-metadata packaging --copy-metadata filelock --copy-metadata numpy --copy-metadata tokenizers --collect-data sacremoses --collect-all transformers

link for notes
pyinstaller/pyinstaller#6210 (reply in thread)

[polm]

Great, glad you figured it out and thanks for reporting back!