error in traning model with metal #81
Assignees
CalvinBackup (Calvin Backup)
Clear assignees
CalvinBackup (Calvin Backup)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: CalvinBackup/DeepFaceLab_MacOS#81
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hi,guys
with edit ".dfl/DeepFaceLab/core/lears/nn.py" line 201 "suggest_all_gpu=True" we can choose metal when traning model
but it still report an error :
Choose one or several GPU idxs (separated by comma).
[CPU] : CPU
[0] : METAL
[0] Which GPU indexes to choose? :
0
Metal device set to: Apple M1
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
Initializing models: 0%| | 0/7 [00:00<?, ?it/s]
Error: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) placed on device Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation:
with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793>
with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index_=-1 requested_device_name_='/device:GPU:0' assigned_device_name_='' resource_device_name_='/device:GPU:0' supported_device_types_=[CPU] possible_devices_=[]
Identity: GPU CPU
Mul: GPU CPU
AddV2: GPU CPU
Sub: GPU CPU
RandomUniform: GPU CPU
Assign: CPU
VariableV2: GPU CPU
Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2)
encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0
encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0
encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0
Assign_2 (Assign) /device:GPU:0
Assign_210 (Assign) /device:GPU:0
(defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61)
]]
Errors may have originated from an input operation.
Input Source operations connected to node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform:
In[0] encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape:
Operation defined at: (most recent call last)
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1579, in get_variable
return get_variable_scope().get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1322, in get_variable
return var_store.get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 578, in get_variable
return _true_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 531, in _true_getter
return self._get_single_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 952, in _get_single_variable
v = variables.VariableV1(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 268, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 213, in _variable_v1_call
return previous_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 206, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2628, in default_variable_creator
return variables.RefVariable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 272, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1651, in init
self._init_from_args(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1794, in _init_from_args
initial_value = initial_value()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 832, in call
return random_ops.random_uniform(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1096, in op_dispatch_handler
return dispatch_target(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 311, in random_uniform
result = gen_random_ops.random_uniform(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in random_uniform
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3697, in _create_op_internal
ret = Operation(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2101, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)
Traceback (most recent call last):
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1380, in do_call
return fn(*args)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1362, in run_fn
self.extend_graph()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1403, in extend_graph
tf_session.ExtendSession(self.session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[]
Identity: GPU CPU
Mul: GPU CPU
AddV2: GPU CPU
Sub: GPU CPU
RandomUniform: GPU CPU
Assign: CPU
VariableV2: GPU CPU
Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2)
encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0
encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0
encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0
Assign_2 (Assign) /device:GPU:0
Assign_210 (Assign) /device:GPU:0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 657, in on_initialize
model.init_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Saveable.py", line 106, in init_weights
nn.init_weights(self.get_weights())
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/ops/__init__.py", line 48, in init_weights
nn.tf_sess.run (ops)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 970, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1193, in run
results = self.do_run(handle, final_targets, final_fetches,
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1373, in do_run
return self.do_call(run_fn, feeds, fetches, targets, options,
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1399, in do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61) placed on device Device assignments active during op 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' creation:
with tf.device(None): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py:1793>
with tf.device(/GPU:0): </Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py:277> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[]
Identity: GPU CPU
Mul: GPU CPU
AddV2: GPU CPU
Sub: GPU CPU
RandomUniform: GPU CPU
Assign: CPU
VariableV2: GPU CPU
Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2)
encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0
encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0
encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0
Assign_2 (Assign) /device:GPU:0
Assign_210 (Assign) /device:GPU:0
(defined at /Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py:61)
]]
Errors may have originated from an input operation.
Input Source operations connected to node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform:
In[0] encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape:
Operation defined at: (most recent call last)
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/opt/homebrew/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1579, in get_variable
return get_variable_scope().get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1322, in get_variable
return var_store.get_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 578, in get_variable
return _true_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 531, in _true_getter
return self._get_single_variable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 952, in _get_single_variable
v = variables.VariableV1(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 268, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 213, in _variable_v1_call
return previous_getter(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 206, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2628, in default_variable_creator
return variables.RefVariable(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 272, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1651, in init
self._init_from_args(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1794, in _init_from_args
initial_value = initial_value()
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 832, in call
return random_ops.random_uniform(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1096, in op_dispatch_handler
return dispatch_target(*args, **kwargs)
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 311, in random_uniform
result = gen_random_ops.random_uniform(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in random_uniform
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 744, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3697, in _create_op_internal
ret = Operation(
File "/Users/wmy/Desktop/tf2/DeepFaceLab_MacOS/.dfl/env/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2101, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)
Same problem
getting this error
`Initializing models: 0%| | 0/5 [00:00<?, ?it/s]
Error: Graph execution error:
Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last):
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform'
Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last):
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self.build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self.build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform'
Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[]
Identity: GPU CPU
Mul: GPU CPU
AddV2: GPU CPU
Sub: GPU CPU
RandomUniform: GPU CPU
Assign: CPU
VariableV2: GPU CPU
Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2)
encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0
encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0
encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0
Assign_2 (Assign) /device:GPU:0
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1617, in get_variable
return get_variable_scope().get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1333, in get_variable
return var_store.get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 589, in get_variable
return _true_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 542, in _true_getter
return self._get_single_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 963, in _get_single_variable
v = variables.VariableV1(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 265, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 210, in _variable_v1_call
return previous_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 203, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2723, in default_variable_creator
return variables.RefVariable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 269, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1648, in init
self._init_from_args(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1791, in _init_from_args
initial_value = initial_value()
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 881, in call
return random_ops.random_uniform(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler
return dispatch_target(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 309, in random_uniform
result = gen_random_ops.random_uniform(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in random_uniform
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal
ret = Operation(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)
Traceback (most recent call last):
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1377, in do_call
return fn(*args)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in run_fn
self.extend_graph()
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1400, in extend_graph
tf_session.ExtendSession(self.session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[]
Identity: GPU CPU
Mul: GPU CPU
AddV2: GPU CPU
Sub: GPU CPU
RandomUniform: GPU CPU
Assign: CPU
VariableV2: GPU CPU
Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2)
encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0
encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0
encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0
Assign_2 (Assign) /device:GPU:0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 657, in on_initialize
model.init_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Saveable.py", line 106, in init_weights
nn.init_weights(self.get_weights())
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/ops/__init__.py", line 48, in init_weights
nn.tf_sess.run (ops)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1370, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1396, in _do_call
raise type(e)(node_def, op, message) # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last):
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform'
Detected at node 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform' defined at (most recent call last):
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self.build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self.build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
Node: 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform'
Cannot assign a device for operation encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform: Could not satisfy explicit device specification '' because the node {{colocation_node encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices=[]
Identity: GPU CPU
Mul: GPU CPU
AddV2: GPU CPU
Sub: GPU CPU
RandomUniform: GPU CPU
Assign: CPU
VariableV2: GPU CPU
Const: GPU CPU
Colocation members, user-requested devices, and framework assigned devices, if any:
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/shape (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/min (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/max (Const)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform (RandomUniform)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/sub (Sub)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/mul (Mul)
encoder/down1/downs_0/conv1/weight/Initializer/random_uniform (AddV2)
encoder/down1/downs_0/conv1/weight (VariableV2) /device:GPU:0
encoder/down1/downs_0/conv1/weight/Assign (Assign) /device:GPU:0
encoder/down1/downs_0/conv1/weight/read (Identity) /device:GPU:0
Assign_2 (Assign) /device:GPU:0
Original stack trace for 'encoder/down1/downs_0/conv1/weight/Initializer/random_uniform/RandomUniform':
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 930, in _bootstrap
self._bootstrap_inner()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 973, in _bootstrap_inner
self.run()
File "/Users/alex/miniforge3/lib/python3.9/threading.py", line 910, in run
self._target(*self._args, **self._kwargs)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/mainscripts/Trainer.py", line 46, in trainerThread
model = models.import_model(model_class_name)(
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/ModelBase.py", line 193, in init
self.on_initialize()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/models/Model_SAEHD/Model.py", line 334, in on_initialize
self.src_dst_saveable_weights = self.encoder.get_weights() + self.inter_AB.get_weights() + self.inter_B.get_weights() + self.decoder.get_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 77, in get_weights
self.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 20, in _build_sub
self.build_sub(sublayer, f"{name}{i}")
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 35, in _build_sub
layer.build()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 65, in build
self._build_sub(v[name],name)
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/models/ModelBase.py", line 33, in _build_sub
layer.build_weights()
File "/Users/alex/Downloads/DeepFaceLab_MacOS-master/.dfl/DeepFaceLab/core/leras/layers/Conv2D.py", line 61, in build_weights
self.weight = tf.get_variable("weight", (self.kernel_size,self.kernel_size,self.in_ch,self.out_ch), dtype=self.dtype, initializer=kernel_initializer, trainable=self.trainable )
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1617, in get_variable
return get_variable_scope().get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 1333, in get_variable
return var_store.get_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 589, in get_variable
return _true_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 542, in _true_getter
return self._get_single_variable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 963, in _get_single_variable
v = variables.VariableV1(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 265, in call
return cls._variable_v1_call(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 210, in _variable_v1_call
return previous_getter(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 203, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variable_scope.py", line 2723, in default_variable_creator
return variables.RefVariable(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 269, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1648, in init
self._init_from_args(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/variables.py", line 1791, in _init_from_args
initial_value = initial_value()
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/init_ops.py", line 881, in call
return random_ops.random_uniform(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1082, in op_dispatch_handler
return dispatch_target(*args, **kwargs)
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/random_ops.py", line 309, in random_uniform
result = gen_random_ops.random_uniform(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/ops/gen_random_ops.py", line 737, in random_uniform
_, _, _op, _outputs = _op_def_library._apply_op_helper(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 740, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3776, in _create_op_internal
ret = Operation(
File "/Users/alex/miniforge3/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2175, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)
`
Wondering if this is our answer https://developer.apple.com/forums/thread/700169?answerId=704851022#704851022
Well got some partial working copy DeepFaceLab_MacOS_M1
@Smiril I am using your fork but cant train my model on GPU . It says tf.reduce_mean has some problem.
Please help me run it on the GPU.
Thanks
@prashantspandey try Metal device
@Smiril I am using M2 macbook pro. Are there any settings I need to change?
@prashantspandey just test and report