GPU issue with multiple Inception V3 trained models
0
$begingroup$
While using multiple (three) trained tensorflow models on python run parallelly as 3 threads (or just 2); I get memory outage but no issue on running each individually on each GPU seperately (2X3 = 6 times) or as per code config below.
GPU config -
GeForce GTX 1060 6GB major totalMemory: 5.93GiB freeMemory: 5.69GiB memoryClockRate(GHz): 1.7715
GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392 =>ignored totalMemory: 3.94GiB freeMemory: 3.89GiB
Individual Model Files' relevant (GPU related) code-
1)d = tf.device('/gpu:0')
config=tf.ConfigProto()
#config.log_device_placement= True
print("SUNGLASSSSSSSSSSSSSSSSSSSS")
#config.gpu_options.per_process_gpu_memory_fraction = 0.3
config=tf.ConfigProto(gpu_options=tf.GPUOptions(visible_device_list='0'))
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
2)
tf.device('/gpu:1')
config=tf.ConfigProto()
#config.log_device_placement= True
print("HATSSSSSSSSSSSSSSSSSS")
config.gpu_options.per_process_gpu_memory_fraction = 0.35
config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True),allow_soft_placement = True)
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
3)
d = '/gpu:1'
config=tf.ConfigProto()
#config.log_device_placement= True
print("HANDSNEARFACEEEEEEEEEEEEE")
config.gpu_options.per_process_gpu_memory_fraction = 0.4
#config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=False),allow_soft_placement = True)
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
Output is
python3 3threads.py
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
2019-02-27 10:47:54.142623: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.142625: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.393343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.394612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.64GiB
2019-02-27 10:47:54.396272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.398889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.59GiB
2019-02-27 10:47:54.554285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555913: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.555967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:54.557271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.557300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.084299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.084339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.084345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.084353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.084752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.085052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fcf4a8e9640>
2019-02-27 10:47:55.102241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.102310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.102318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.102323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.102749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.103170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fe04e1e6500>
2019-02-27 10:47:55.434425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.434480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.434488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.434493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.434521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.434836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.434975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.444271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.444283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.444292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.444301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.444619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.471610: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 5.17G (5557321728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.472972: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.66G (5001589248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.474334: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.19G (4501430272 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.475648: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.77G (4051287040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.477000: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.40G (3646158336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.478323: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.06G (3281542400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.479630: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.75G (2953388032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.481035: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.47G (2658049280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.483432: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.23G (2392244224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.485884: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.00G (2153019648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.488176: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.80G (1937717760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.490135: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.62G (1743945984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.491335: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.46G (1569551360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.492490: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.32G (1412596224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.493622: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.18G (1271336704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.494845: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.07G (1144203008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.496096: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 982.08M (1029782784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.497218: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 883.87M (926804480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.498403: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 795.48M (834124032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.499495: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 715.93M (750711808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
SUNGLASSSSSSSSSSSSSSSSSSSS
SUNGLASSSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.500682: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 644.34M (675640832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.501167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-27 10:47:55.501225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.501241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-02-27 10:47:55.501255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-02-27 10:47:55.501466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.501831: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 579.91M (608076800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.502756: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 521.92M (547269120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.503654: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 469.72M (492542208 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.504581: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 422.75M (443288064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.505485: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 380.48M (398959360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.506384: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 342.43M (359063552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.507267: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 308.19M (323157248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
HATSSSSSSSSSSSSSSSSSS
HATSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.515435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.515476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.515484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.515492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.515498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.515754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.515837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:56.361510: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.365800: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.370375: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.371931: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.375127: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.378651: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "/home/facialstats/Downloads/sunglass/label_image.py", line 195, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/sunglass/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/sunglass/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
File "/home/facialstats/Downloads/hats/label_image.py", line 194, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/hats/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/hats/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Even 2 files alone don't run.
tensorflow gpu inception
asked yesterday
MikeMike
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
0
$begingroup$
While using multiple (three) trained tensorflow models on python run parallelly as 3 threads (or just 2); I get memory outage but no issue on running each individually on each GPU seperately (2X3 = 6 times) or as per code config below.
GPU config -
GeForce GTX 1060 6GB major totalMemory: 5.93GiB freeMemory: 5.69GiB memoryClockRate(GHz): 1.7715
GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392 =>ignored totalMemory: 3.94GiB freeMemory: 3.89GiB
Individual Model Files' relevant (GPU related) code-
1)d = tf.device('/gpu:0')
config=tf.ConfigProto()
#config.log_device_placement= True
print("SUNGLASSSSSSSSSSSSSSSSSSSS")
#config.gpu_options.per_process_gpu_memory_fraction = 0.3
config=tf.ConfigProto(gpu_options=tf.GPUOptions(visible_device_list='0'))
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
2)
tf.device('/gpu:1')
config=tf.ConfigProto()
#config.log_device_placement= True
print("HATSSSSSSSSSSSSSSSSSS")
config.gpu_options.per_process_gpu_memory_fraction = 0.35
config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True),allow_soft_placement = True)
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
3)
d = '/gpu:1'
config=tf.ConfigProto()
#config.log_device_placement= True
print("HANDSNEARFACEEEEEEEEEEEEE")
config.gpu_options.per_process_gpu_memory_fraction = 0.4
#config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=False),allow_soft_placement = True)
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
Output is
python3 3threads.py
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
2019-02-27 10:47:54.142623: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.142625: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.393343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.394612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.64GiB
2019-02-27 10:47:54.396272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.398889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.59GiB
2019-02-27 10:47:54.554285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555913: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.555967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:54.557271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.557300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.084299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.084339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.084345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.084353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.084752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.085052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fcf4a8e9640>
2019-02-27 10:47:55.102241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.102310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.102318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.102323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.102749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.103170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fe04e1e6500>
2019-02-27 10:47:55.434425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.434480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.434488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.434493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.434521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.434836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.434975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.444271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.444283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.444292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.444301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.444619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.471610: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 5.17G (5557321728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.472972: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.66G (5001589248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.474334: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.19G (4501430272 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.475648: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.77G (4051287040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.477000: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.40G (3646158336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.478323: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.06G (3281542400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.479630: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.75G (2953388032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.481035: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.47G (2658049280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.483432: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.23G (2392244224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.485884: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.00G (2153019648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.488176: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.80G (1937717760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.490135: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.62G (1743945984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.491335: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.46G (1569551360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.492490: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.32G (1412596224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.493622: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.18G (1271336704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.494845: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.07G (1144203008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.496096: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 982.08M (1029782784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.497218: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 883.87M (926804480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.498403: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 795.48M (834124032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.499495: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 715.93M (750711808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
SUNGLASSSSSSSSSSSSSSSSSSSS
SUNGLASSSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.500682: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 644.34M (675640832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.501167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-27 10:47:55.501225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.501241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-02-27 10:47:55.501255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-02-27 10:47:55.501466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.501831: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 579.91M (608076800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.502756: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 521.92M (547269120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.503654: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 469.72M (492542208 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.504581: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 422.75M (443288064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.505485: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 380.48M (398959360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.506384: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 342.43M (359063552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.507267: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 308.19M (323157248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
HATSSSSSSSSSSSSSSSSSS
HATSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.515435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.515476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.515484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.515492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.515498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.515754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.515837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:56.361510: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.365800: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.370375: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.371931: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.375127: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.378651: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "/home/facialstats/Downloads/sunglass/label_image.py", line 195, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/sunglass/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/sunglass/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
File "/home/facialstats/Downloads/hats/label_image.py", line 194, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/hats/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/hats/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Even 2 files alone don't run.
tensorflow gpu inception
asked yesterday
MikeMike
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
0
0
0
$begingroup$
While using multiple (three) trained tensorflow models on python run parallelly as 3 threads (or just 2); I get memory outage but no issue on running each individually on each GPU seperately (2X3 = 6 times) or as per code config below.
GPU config -
GeForce GTX 1060 6GB major totalMemory: 5.93GiB freeMemory: 5.69GiB memoryClockRate(GHz): 1.7715
GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392 =>ignored totalMemory: 3.94GiB freeMemory: 3.89GiB
Individual Model Files' relevant (GPU related) code-
1)d = tf.device('/gpu:0')
config=tf.ConfigProto()
#config.log_device_placement= True
print("SUNGLASSSSSSSSSSSSSSSSSSSS")
#config.gpu_options.per_process_gpu_memory_fraction = 0.3
config=tf.ConfigProto(gpu_options=tf.GPUOptions(visible_device_list='0'))
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
2)
tf.device('/gpu:1')
config=tf.ConfigProto()
#config.log_device_placement= True
print("HATSSSSSSSSSSSSSSSSSS")
config.gpu_options.per_process_gpu_memory_fraction = 0.35
config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True),allow_soft_placement = True)
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
3)
d = '/gpu:1'
config=tf.ConfigProto()
#config.log_device_placement= True
print("HANDSNEARFACEEEEEEEEEEEEE")
config.gpu_options.per_process_gpu_memory_fraction = 0.4
#config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=False),allow_soft_placement = True)
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
Output is
python3 3threads.py
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
2019-02-27 10:47:54.142623: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.142625: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.393343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.394612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.64GiB
2019-02-27 10:47:54.396272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.398889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.59GiB
2019-02-27 10:47:54.554285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555913: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.555967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:54.557271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.557300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.084299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.084339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.084345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.084353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.084752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.085052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fcf4a8e9640>
2019-02-27 10:47:55.102241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.102310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.102318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.102323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.102749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.103170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fe04e1e6500>
2019-02-27 10:47:55.434425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.434480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.434488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.434493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.434521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.434836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.434975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.444271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.444283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.444292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.444301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.444619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.471610: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 5.17G (5557321728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.472972: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.66G (5001589248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.474334: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.19G (4501430272 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.475648: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.77G (4051287040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.477000: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.40G (3646158336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.478323: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.06G (3281542400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.479630: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.75G (2953388032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.481035: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.47G (2658049280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.483432: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.23G (2392244224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.485884: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.00G (2153019648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.488176: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.80G (1937717760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.490135: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.62G (1743945984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.491335: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.46G (1569551360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.492490: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.32G (1412596224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.493622: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.18G (1271336704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.494845: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.07G (1144203008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.496096: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 982.08M (1029782784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.497218: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 883.87M (926804480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.498403: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 795.48M (834124032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.499495: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 715.93M (750711808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
SUNGLASSSSSSSSSSSSSSSSSSSS
SUNGLASSSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.500682: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 644.34M (675640832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.501167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-27 10:47:55.501225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.501241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-02-27 10:47:55.501255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-02-27 10:47:55.501466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.501831: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 579.91M (608076800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.502756: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 521.92M (547269120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.503654: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 469.72M (492542208 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.504581: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 422.75M (443288064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.505485: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 380.48M (398959360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.506384: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 342.43M (359063552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.507267: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 308.19M (323157248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
HATSSSSSSSSSSSSSSSSSS
HATSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.515435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.515476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.515484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.515492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.515498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.515754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.515837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:56.361510: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.365800: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.370375: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.371931: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.375127: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.378651: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "/home/facialstats/Downloads/sunglass/label_image.py", line 195, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/sunglass/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/sunglass/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
File "/home/facialstats/Downloads/hats/label_image.py", line 194, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/hats/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/hats/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Even 2 files alone don't run.
tensorflow gpu inception
asked yesterday
MikeMike
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
While using multiple (three) trained tensorflow models on python run parallelly as 3 threads (or just 2); I get memory outage but no issue on running each individually on each GPU seperately (2X3 = 6 times) or as per code config below.
GPU config -
GeForce GTX 1060 6GB major totalMemory: 5.93GiB freeMemory: 5.69GiB memoryClockRate(GHz): 1.7715
GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392 =>ignored totalMemory: 3.94GiB freeMemory: 3.89GiB
Individual Model Files' relevant (GPU related) code-
1)d = tf.device('/gpu:0')
config=tf.ConfigProto()
#config.log_device_placement= True
print("SUNGLASSSSSSSSSSSSSSSSSSSS")
#config.gpu_options.per_process_gpu_memory_fraction = 0.3
config=tf.ConfigProto(gpu_options=tf.GPUOptions(visible_device_list='0'))
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
2)
tf.device('/gpu:1')
config=tf.ConfigProto()
#config.log_device_placement= True
print("HATSSSSSSSSSSSSSSSSSS")
config.gpu_options.per_process_gpu_memory_fraction = 0.35
config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True),allow_soft_placement = True)
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
3)
d = '/gpu:1'
config=tf.ConfigProto()
#config.log_device_placement= True
print("HANDSNEARFACEEEEEEEEEEEEE")
config.gpu_options.per_process_gpu_memory_fraction = 0.4
#config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=False),allow_soft_placement = True)
with tf.device(d):
with tf.Session(graph=graph, config=config) as sess:
results = sess.run(output_operation.outputs[0], {
input_operation.outputs[0]: t
})
results = np.squeeze(results)
Output is
python3 3threads.py
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
/usr/local/lib/python2.7/dist-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
warnings.warn(warning, RequestsDependencyWarning)
2019-02-27 10:47:54.142623: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.142625: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-02-27 10:47:54.393343: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.394612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.64GiB
2019-02-27 10:47:54.396272: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.398889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
pciBusID: 0000:06:00.0
totalMemory: 5.93GiB freeMemory: 5.59GiB
2019-02-27 10:47:54.554285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555913: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-02-27 10:47:54.555927: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.555967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:54.557271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.392
pciBusID: 0000:05:00.0
totalMemory: 3.94GiB freeMemory: 3.85GiB
2019-02-27 10:47:54.557300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.084299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.084339: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.084345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.084353: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.084752: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.085052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fcf4a8e9640>
2019-02-27 10:47:55.102241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.102310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.102318: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.102323: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.102749: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.103170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
<generator object <genexpr> at 0x7fe04e1e6500>
2019-02-27 10:47:55.434425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.434480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.434488: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.434493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.434521: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.434836: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.434975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3516 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.444271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.444283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.444292: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.444301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.444619: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.444796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:55.471610: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 5.17G (5557321728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.472972: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.66G (5001589248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.474334: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 4.19G (4501430272 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.475648: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.77G (4051287040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.477000: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.40G (3646158336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.478323: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 3.06G (3281542400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.479630: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.75G (2953388032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.481035: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.47G (2658049280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.483432: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.23G (2392244224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.485884: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.00G (2153019648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.488176: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.80G (1937717760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.490135: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.62G (1743945984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.491335: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.46G (1569551360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.492490: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.32G (1412596224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.493622: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.18G (1271336704 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.494845: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 1.07G (1144203008 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.496096: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 982.08M (1029782784 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.497218: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 883.87M (926804480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.498403: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 795.48M (834124032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.499495: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 715.93M (750711808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
SUNGLASSSSSSSSSSSSSSSSSSSS
SUNGLASSSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.500682: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 644.34M (675640832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.501167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-02-27 10:47:55.501225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.501241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0
2019-02-27 10:47:55.501255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N
2019-02-27 10:47:55.501466: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.501831: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 579.91M (608076800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.502756: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 521.92M (547269120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.503654: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 469.72M (492542208 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.504581: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 422.75M (443288064 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.505485: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 380.48M (398959360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.506384: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 342.43M (359063552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-02-27 10:47:55.507267: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 308.19M (323157248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
HATSSSSSSSSSSSSSSSSSS
HATSSSSSSSSSSSSSSSSSS
2019-02-27 10:47:55.515435: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1
2019-02-27 10:47:55.515476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-02-27 10:47:55.515484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 1
2019-02-27 10:47:55.515492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N N
2019-02-27 10:47:55.515498: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1: N N
2019-02-27 10:47:55.515754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5299 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:06:00.0, compute capability: 6.1)
2019-02-27 10:47:55.515837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 3512 MB memory) -> physical GPU (device: 1, name: GeForce GTX 1050 Ti, pci bus id: 0000:05:00.0, compute capability: 6.1)
2019-02-27 10:47:56.361510: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.365800: E tensorflow/stream_executor/cuda/cuda_blas.cc:464] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-02-27 10:47:56.370375: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.371931: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.375127: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2019-02-27 10:47:56.378651: E tensorflow/stream_executor/cuda/cuda_dnn.cc:373] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):
File "/home/facialstats/Downloads/sunglass/label_image.py", line 195, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/sunglass/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/sunglass/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/sunglass/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Traceback (most recent call last):
File "/home/facialstats/Downloads/hats/label_image.py", line 194, in <module>
input_operation.outputs[0]: t
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D', defined at:
File "/home/facialstats/Downloads/hats/label_image.py", line 169, in <module>
graph = load_graph(model_file)
File "/home/facialstats/Downloads/hats/label_image.py", line 58, in load_graph
tf.import_graph_def(graph_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
_ProcessNewOps(graph)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 234, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3440, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3299, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
self._traceback = tf_stack.extract_stack()
UnknownError (see above for traceback): Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D (defined at /home/facialstats/Downloads/hats/label_image.py:58) = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="VALID", strides=[1, 1, 2, 2], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](import/module_apply_default/InceptionV3/InceptionV3/Conv2d_1a_3x3/Conv2D-0-TransposeNHWCToNCHW-LayoutOptimizer, import/module/InceptionV3/Conv2d_1a_3x3/weights)]]
[[{{node import/final_result/_3}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1005_import/final_result", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Even 2 files alone don't run.
tensorflow gpu inception
tensorflow gpu inception
asked yesterday
MikeMike
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
MikeMike
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 15 hours ago
Mike
asked yesterday
MikeMike
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked yesterday
MikeMike
11
asked yesterday
MikeMike
11
11
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Mike is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "557"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Mike is a new contributor. Be nice, and check out our Code of Conduct.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46285%2fgpu-issue-with-multiple-inception-v3-trained-models%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Mike is a new contributor. Be nice, and check out our Code of Conduct.
draft saved
draft discarded
Mike is a new contributor. Be nice, and check out our Code of Conduct.
Mike is a new contributor. Be nice, and check out our Code of Conduct.
Mike is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f46285%2fgpu-issue-with-multiple-inception-v3-trained-models%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
