CalvinBackup/DeepFaceLab_MacOS
1
0
Fork 0
Code Issues 21 Pull Requests 8 Actions Packages Projects Releases 2 Wiki Activity

How much GPU ram is available when I train model in M1 Macbook? #37

New Issue
Closed
opened 2021-10-13 19:35:50 +02:00 by torres-c · 12 comments
torres-c commented 2021-10-13 19:35:50 +02:00 (Migrated from github.com)
Copy Link
Copy Source

How much GPU ram is available when I train model in M1 Macbook?

How much GPU ram is available when I train model in M1 Macbook?
chychkan commented 2021-10-22 02:35:09 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Hi @Dfaker-HK!
I'm not familiar enough with latest Mac's hardware, so can't give you an expert advise, but I find this discussion quite helpful as it gives couple good links for further reading.

As far as I understand, in the new Mac's CPU and GPU have shared RAM, so I would expect that GPU can use all the free RAM that you have on your laptop given that it runs the OS and all the processes.

Hi @Dfaker-HK! I'm not familiar enough with latest Mac's hardware, so can't give you an expert advise, but I find [this discussion](https://discussions.apple.com/thread/252725837) quite helpful as it gives couple good links for further reading. As far as I understand, in the new Mac's CPU and GPU have shared RAM, so I would expect that GPU can use all the free RAM that you have on your laptop given that it runs the OS and all the processes.
bufo24 commented 2021-10-23 16:05:51 +02:00 (Migrated from github.com)
Copy Link
Copy Source

May I ask how you did get it running on M1? I am struggling with the python libraries.

May I ask how you did get it running on M1? I am struggling with the python libraries.
👍 1
chychkan commented 2021-10-24 06:49:13 +02:00 (Migrated from github.com)
Copy Link
Copy Source

May I ask how you did get it running on M1? I am struggling with the python libraries.

@bufo24 I didn't get chance to test it on M1. Could you please open a separate issue and provide details of what is your Python version and what's the error output you're getting?

> May I ask how you did get it running on M1? I am struggling with the python libraries. @bufo24 I didn't get chance to test it on M1. Could you please open a separate issue and provide details of what is your Python version and what's the error output you're getting?
torres-c commented 2021-10-24 13:12:23 +02:00 (Migrated from github.com)
Copy Link
Copy Source

May I ask how you did get it running on M1? I am struggling with the python libraries.

@bufo24 I didn't get chance to test it on M1. Could you please open a separate issue and provide details of what is your Python version and what's the error output you're getting?

I am not getting the error actually
I just plan to buy the new m1 macbook and use it to train the model 7x24 as my 3090 is too loud that I can’t let it run when I am sleeping.
And I want to know what model can I run on it.

> > May I ask how you did get it running on M1? I am struggling with the python libraries. > > @bufo24 I didn't get chance to test it on M1. Could you please open a separate issue and provide details of what is your Python version and what's the error output you're getting? I am not getting the error actually I just plan to buy the new m1 macbook and use it to train the model 7x24 as my 3090 is too loud that I can’t let it run when I am sleeping. And I want to know what model can I run on it.
github-actions[bot] commented 2021-12-12 15:50:02 +01:00 (Migrated from github.com)
Copy Link
Copy Source

The issue is tagged as "stale issue" due to no activity in 30 days. If there is no activity for 15 more days, the issue will be closed.

The issue is tagged as "stale issue" due to no activity in 30 days. If there is no activity for 15 more days, the issue will be closed.
mateuspestana commented 2022-01-11 04:16:37 +01:00 (Migrated from github.com)
Copy Link
Copy Source

I am trying to run on my MBP M1 Pro and the lack of tensorflow version for the M1 is killing me! Did anyone solve that? All I get is:


ERROR: Could not find a version that satisfies the requirement tensorflow==2.7.0 (from versions: none)
ERROR: No matching distribution found for tensorflow==2.7.0

Edit:

My workaround was: installing the tensorflow-macos and tensorflow-metal, and for PyQt5, I installed from brew (brew install pyqt@5), copied the files inside site-packages from the Homebrew/Caskroom folders and paste it on the env folders of DeepfacelabMacOS. Now it works.

I am trying to run on my MBP M1 Pro and the lack of tensorflow version for the M1 is killing me! Did anyone solve that? All I get is: ``` ERROR: Could not find a version that satisfies the requirement tensorflow==2.7.0 (from versions: none) ERROR: No matching distribution found for tensorflow==2.7.0 ``` Edit: My workaround was: installing the tensorflow-macos and tensorflow-metal, and for PyQt5, I installed from brew (`brew install pyqt@5`), copied the files inside site-packages from the Homebrew/Caskroom folders and paste it on the env folders of DeepfacelabMacOS. Now it works.
chychkan commented 2022-01-11 20:10:32 +01:00 (Migrated from github.com)
Copy Link
Copy Source

@mateuspestana what Python version do you have? Currently only Python up to 3.9 (included) is supported. The 3.10 is not supported yet. If you have 3.10, you can try using pyenv to switch between 3.9 and 3.10. When you switch to 3.9, make sure that both python and virtualenv point at the same version (check with python -V and virtualenv --version).

@mateuspestana what Python version do you have? Currently only Python up to `3.9` (included) is supported. The `3.10` is not supported yet. If you have `3.10`, you can try using [pyenv](https://github.com/pyenv/pyenv) to switch between `3.9` and `3.10`. When you switch to `3.9`, make sure that both `python` and `virtualenv` point at the same version (check with `python -V` and `virtualenv --version`).
mateuspestana commented 2022-01-11 20:13:04 +01:00 (Migrated from github.com)
Copy Link
Copy Source

I'm on 3.9!

I'm on 3.9!
chychkan commented 2022-01-11 20:20:43 +01:00 (Migrated from github.com)
Copy Link
Copy Source

@mateuspestana Hmm, interesting. Unfortunately, I don't have M1 to troubleshoot it myself. Do you mind sharing full console output just in case? Would be great if you also remove the .dfl/ dir to make sure you're starting from scratch.

@mateuspestana Hmm, interesting. Unfortunately, I don't have M1 to troubleshoot it myself. Do you mind sharing full console output just in case? Would be great if you also remove the `.dfl/` dir to make sure you're starting from scratch.
mateuspestana commented 2022-01-11 21:04:06 +01:00 (Migrated from github.com)
Copy Link
Copy Source
❯ ./0_setup.sh
Cloning DeepFaceLab
Cloning into '.dfl/DeepFaceLab'...
remote: Enumerating objects: 247, done.
remote: Counting objects: 100% (247/247), done.
remote: Compressing objects: 100% (223/223), done.
remote: Total 247 (delta 24), reused 141 (delta 15), pack-reused 0
Receiving objects: 100% (247/247), 253.22 MiB | 21.25 MiB/s, done.
Resolving deltas: 100% (24/24), done.
Updating files: 100% (211/211), done.
created virtual environment CPython3.9.9.final.0-64 in 181ms
  creator CPython3Posix(dest=/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mateuspestana/Library/Application Support/virtualenv)
    added seed packages: pip==21.3.1, setuptools==60.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Using requirements_3.9.txt for Python 3.9.9
Collecting numpy
  Using cached numpy-1.22.0-cp39-cp39-macosx_11_0_arm64.whl (12.8 MB)
Collecting opencv-contrib-python-headless==4.5.5.62
  Using cached opencv_contrib_python_headless-4.5.5.62-cp37-abi3-macosx_11_0_arm64.whl (36.1 MB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-1-cp39-cp39-macosx_12_0_arm64.whl (27.0 MB)
Collecting tensorflow-macos
  Using cached tensorflow_macos-2.7.0-cp39-cp39-macosx_11_0_arm64.whl (179.0 MB)
Collecting tensorflow-metal
  Using cached tensorflow_metal-0.3.0-cp39-cp39-macosx_11_0_arm64.whl (1.2 MB)
ERROR: Could not find a version that satisfies the requirement tensorflow==2.7.0 (from versions: none)
ERROR: No matching distribution found for tensorflow==2.7.0
❯ ./0_setup.sh
Using requirements_3.9.txt for Python 3.9.9
Collecting numpy
  Using cached numpy-1.22.0-cp39-cp39-macosx_11_0_arm64.whl (12.8 MB)
Collecting opencv-contrib-python-headless==4.5.5.62
  Using cached opencv_contrib_python_headless-4.5.5.62-cp37-abi3-macosx_11_0_arm64.whl (36.1 MB)
Collecting scipy==1.7.3
  Using cached scipy-1.7.3-1-cp39-cp39-macosx_12_0_arm64.whl (27.0 MB)
Collecting tensorflow-macos
  Using cached tensorflow_macos-2.7.0-cp39-cp39-macosx_11_0_arm64.whl (179.0 MB)
Collecting tensorflow-metal
  Using cached tensorflow_metal-0.3.0-cp39-cp39-macosx_11_0_arm64.whl (1.2 MB)
Collecting colorama==0.4.4
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting tqdm==4.62.3
  Using cached tqdm-4.62.3-py2.py3-none-any.whl (76 kB)
Collecting ffmpeg-python==0.2.0
  Using cached ffmpeg_python-0.2.0-py3-none-any.whl (25 kB)
Collecting Pillow==8.4.0
  Using cached Pillow-8.4.0-cp39-cp39-macosx_11_0_arm64.whl (2.8 MB)
Collecting scikit-image
  Using cached scikit_image-0.19.1-cp39-cp39-macosx_12_0_arm64.whl (12.2 MB)
Collecting h5py==3.6.0
  Using cached h5py-3.6.0-cp39-cp39-macosx_11_0_arm64.whl
Collecting numexpr==2.8.1
  Using cached numexpr-2.8.1-cp39-cp39-macosx_12_0_arm64.whl
Collecting PyQt5==5.15.4
  Using cached PyQt5-5.15.4.tar.gz (3.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/bin/python /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/tmp220idrr2
       cwd: /private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-install-_7sl96sx/pyqt5_68db5ba4f5f847b5ba24c1bcb5d6a612
  Complete output (24 lines):
  Querying qmake about your Qt installation...
  /opt/homebrew/bin/qmake -query
  Traceback (most recent call last):
    File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
      hook = backend.prepare_metadata_for_build_wheel
  AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-build-env-8opurhlj/overlay/lib/python3.9/site-packages/sipbuild/api.py", line 51, in build_wheel
      project = AbstractProject.bootstrap('pep517')
    File "/private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-build-env-8opurhlj/overlay/lib/python3.9/site-packages/sipbuild/abstract_project.py", line 83, in bootstrap
      project.setup(pyproject, tool, tool_description)
    File "/private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-build-env-8opurhlj/overlay/lib/python3.9/site-packages/sipbuild/project.py", line 603, in setup
      self.update(tool)
    File "project.py", line 160, in update
  sipbuild.exceptions.UserException
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/8e/a4/d5e4bf99dd50134c88b95e926d7b81aad2473b47fde5e3e4eac2c69a8942/PyQt5-5.15.4.tar.gz#sha256=2a69597e0dd11caabe75fae133feca66387819fc9bc050f547e5551bce97e5be (from https://pypi.org/simple/pyqt5/) (requires-python:>=3.6). Command errored out with exit status 1: /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/bin/python /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/tmp220idrr2 Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement PyQt5==5.15.4 (from versions: 5.14.0, 5.14.1, 5.14.2, 5.15.0, 5.15.1, 5.15.2, 5.15.3, 5.15.4, 5.15.5, 5.15.6)
ERROR: No matching distribution found for PyQt5==5.15.4

What I do after that so it works: I install PyQt5 from brew and put the files inside the env (and remove the line from the requirements and run the rest), so when it runs, it gets "sees" version and can import.

``` ❯ ./0_setup.sh Cloning DeepFaceLab Cloning into '.dfl/DeepFaceLab'... remote: Enumerating objects: 247, done. remote: Counting objects: 100% (247/247), done. remote: Compressing objects: 100% (223/223), done. remote: Total 247 (delta 24), reused 141 (delta 15), pack-reused 0 Receiving objects: 100% (247/247), 253.22 MiB | 21.25 MiB/s, done. Resolving deltas: 100% (24/24), done. Updating files: 100% (211/211), done. created virtual environment CPython3.9.9.final.0-64 in 181ms creator CPython3Posix(dest=/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env, clear=False, no_vcs_ignore=False, global=False) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/mateuspestana/Library/Application Support/virtualenv) added seed packages: pip==21.3.1, setuptools==60.2.0, wheel==0.37.1 activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator Using requirements_3.9.txt for Python 3.9.9 Collecting numpy Using cached numpy-1.22.0-cp39-cp39-macosx_11_0_arm64.whl (12.8 MB) Collecting opencv-contrib-python-headless==4.5.5.62 Using cached opencv_contrib_python_headless-4.5.5.62-cp37-abi3-macosx_11_0_arm64.whl (36.1 MB) Collecting scipy==1.7.3 Using cached scipy-1.7.3-1-cp39-cp39-macosx_12_0_arm64.whl (27.0 MB) Collecting tensorflow-macos Using cached tensorflow_macos-2.7.0-cp39-cp39-macosx_11_0_arm64.whl (179.0 MB) Collecting tensorflow-metal Using cached tensorflow_metal-0.3.0-cp39-cp39-macosx_11_0_arm64.whl (1.2 MB) ERROR: Could not find a version that satisfies the requirement tensorflow==2.7.0 (from versions: none) ERROR: No matching distribution found for tensorflow==2.7.0 ❯ ./0_setup.sh Using requirements_3.9.txt for Python 3.9.9 Collecting numpy Using cached numpy-1.22.0-cp39-cp39-macosx_11_0_arm64.whl (12.8 MB) Collecting opencv-contrib-python-headless==4.5.5.62 Using cached opencv_contrib_python_headless-4.5.5.62-cp37-abi3-macosx_11_0_arm64.whl (36.1 MB) Collecting scipy==1.7.3 Using cached scipy-1.7.3-1-cp39-cp39-macosx_12_0_arm64.whl (27.0 MB) Collecting tensorflow-macos Using cached tensorflow_macos-2.7.0-cp39-cp39-macosx_11_0_arm64.whl (179.0 MB) Collecting tensorflow-metal Using cached tensorflow_metal-0.3.0-cp39-cp39-macosx_11_0_arm64.whl (1.2 MB) Collecting colorama==0.4.4 Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB) Collecting tqdm==4.62.3 Using cached tqdm-4.62.3-py2.py3-none-any.whl (76 kB) Collecting ffmpeg-python==0.2.0 Using cached ffmpeg_python-0.2.0-py3-none-any.whl (25 kB) Collecting Pillow==8.4.0 Using cached Pillow-8.4.0-cp39-cp39-macosx_11_0_arm64.whl (2.8 MB) Collecting scikit-image Using cached scikit_image-0.19.1-cp39-cp39-macosx_12_0_arm64.whl (12.2 MB) Collecting h5py==3.6.0 Using cached h5py-3.6.0-cp39-cp39-macosx_11_0_arm64.whl Collecting numexpr==2.8.1 Using cached numexpr-2.8.1-cp39-cp39-macosx_12_0_arm64.whl Collecting PyQt5==5.15.4 Using cached PyQt5-5.15.4.tar.gz (3.3 MB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... error ERROR: Command errored out with exit status 1: command: /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/bin/python /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/tmp220idrr2 cwd: /private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-install-_7sl96sx/pyqt5_68db5ba4f5f847b5ba24c1bcb5d6a612 Complete output (24 lines): Querying qmake about your Qt installation... /opt/homebrew/bin/qmake -query Traceback (most recent call last): File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel hook = backend.prepare_metadata_for_build_wheel AttributeError: module 'sipbuild.api' has no attribute 'prepare_metadata_for_build_wheel' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module> main() File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel whl_basename = backend.build_wheel(metadata_directory, config_settings) File "/private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-build-env-8opurhlj/overlay/lib/python3.9/site-packages/sipbuild/api.py", line 51, in build_wheel project = AbstractProject.bootstrap('pep517') File "/private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-build-env-8opurhlj/overlay/lib/python3.9/site-packages/sipbuild/abstract_project.py", line 83, in bootstrap project.setup(pyproject, tool, tool_description) File "/private/var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/pip-build-env-8opurhlj/overlay/lib/python3.9/site-packages/sipbuild/project.py", line 603, in setup self.update(tool) File "project.py", line 160, in update sipbuild.exceptions.UserException ---------------------------------------- WARNING: Discarding https://files.pythonhosted.org/packages/8e/a4/d5e4bf99dd50134c88b95e926d7b81aad2473b47fde5e3e4eac2c69a8942/PyQt5-5.15.4.tar.gz#sha256=2a69597e0dd11caabe75fae133feca66387819fc9bc050f547e5551bce97e5be (from https://pypi.org/simple/pyqt5/) (requires-python:>=3.6). Command errored out with exit status 1: /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/bin/python /Users/mateuspestana/Downloads/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/ns/bgqhy0js3djctln1yxdwt23r0000gn/T/tmp220idrr2 Check the logs for full command output. ERROR: Could not find a version that satisfies the requirement PyQt5==5.15.4 (from versions: 5.14.0, 5.14.1, 5.14.2, 5.15.0, 5.15.1, 5.15.2, 5.15.3, 5.15.4, 5.15.5, 5.15.6) ERROR: No matching distribution found for PyQt5==5.15.4 ``` What I do after that so it works: I install PyQt5 from brew and put the files inside the env (and remove the line from the requirements and run the rest), so when it runs, it gets "sees" version and can import.
github-actions[bot] commented 2022-02-11 15:49:49 +01:00 (Migrated from github.com)
Copy Link
Copy Source

The issue is tagged as "stale issue" due to no activity in 30 days. If there is no activity for 15 more days, the issue will be closed.

The issue is tagged as "stale issue" due to no activity in 30 days. If there is no activity for 15 more days, the issue will be closed.
github-actions[bot] commented 2022-03-28 16:49:56 +02:00 (Migrated from github.com)
Copy Link
Copy Source

Closing this issue due to no activity in last 45 days

Closing this issue due to no activity in last 45 days
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something isn't working
 dependencies
Pull requests that update a dependency file
 documentation
Improvements or additions to documentation
 duplicate
This issue or pull request already exists
 enhancement
New feature or request
 good first issue
Good for newcomers
 help wanted
Extra attention is needed
 invalid
This doesn't seem right
 placeholder
Keeping open for more changes
 question
Further information is requested
 stale issue
No activity in 30 days
 wontfix
This will not be worked on
No labels stale issue
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
CalvinBackup (Calvin Backup)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: CalvinBackup/DeepFaceLab_MacOS#37