GPU memory question
Hello everyone,
I am sorry if this is a stupid question. So I got a RTX3090, and I am trying to simulate a high resolution dam-break case. I am using the precompiled Windows-version of DualSPHhysics with the latest Nvidia drivers (531.18 Nvidia game ready driver/528.49 Nvidia studio driver).
The GPU has a total memory of 32 GB. However, if I am trying to decrease the particle distance to about 2.5 mm, the solver is giving me a: "Failed GPU memory allocation. (CUDA error 2 (out of memory))". Nvidiainspector and the Windows task manager are showing me a memory usage of about 10 GB for a simulation with a particle distance of 3 mm. Therefore, I am assuming that a marginally smaller particle distance should also work. However, if I decrease the particle distance slightly (e.g. to 2.9 mm) I am also given the error from above. CUDA is showing a usage of about 95% in the Windows task manager for a simulation with 3 mm.
So I do not understand why I cannot decrease the particle distance by a greater amount. Am I misunderstanding the usage of memory here? Is there a different memory limit for CUDA? Is there some special Windows setting to use all the memory? I really do not understand why the Solver is complaining, as I can use the full memory on my laptop with a Linux environment and a GeForce RTX 3050 Laptop.
Thanks for your help in advance
Felix
Debug Trace
| Notice |
rich is deprecated. Use FormatService::renderHtml($content, Formats\RichFormat::FORMAT_KEY) instead.
#0 [internal function]: gdn_ErrorHandler(16384, 'rich is depreca...', '/var/www/forums...', 950, Array)
#1 /var/www/forums-dual-sphysics-org/library/core/functions.general.php(950): trigger_error('rich is depreca...', 16384)
#2 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1729): deprecated('rich', 'FormatService::...')
#3 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1479): Gdn_Format::rich('[{"insert":"Hel...')
#4 /var/www/forums-dual-sphysics-org/applications/vanilla/controllers/class.discussioncontroller.php(197): Gdn_Format::to('[{"insert":"Hel...', 'Rich')
#5 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(862): DiscussionController->index('2519', 'gpu-memory-ques...', '')
#6 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(279): Gdn_Dispatcher->dispatchController(Object(Gdn_Request), Array)
#7 /var/www/forums-dual-sphysics-org/index.php(29): Gdn_Dispatcher->dispatch()
#8 {main} |
| Notice |
rich is deprecated. Use FormatService::renderHtml($content, Formats\RichFormat::FORMAT_KEY) instead.
#0 [internal function]: gdn_ErrorHandler(16384, 'rich is depreca...', '/var/www/forums...', 950, Array)
#1 /var/www/forums-dual-sphysics-org/library/core/functions.general.php(950): trigger_error('rich is depreca...', 16384)
#2 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1729): deprecated('rich', 'FormatService::...')
#3 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1479): Gdn_Format::rich('[{"insert":"Hel...')
#4 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(24): Gdn_Format::to('[{"insert":"Hel...', 'Rich')
#5 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/discussion.php(89): formatBody(Object(stdClass))
#6 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/index.php(31): include('/var/www/forums...')
#7 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(778): include('/var/www/forums...')
#8 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(1382): Gdn_Controller->fetchView('', false, false)
#9 /var/www/forums-dual-sphysics-org/library/core/class.pluggable.php(217): Gdn_Controller->xRender()
#10 /var/www/forums-dual-sphysics-org/applications/vanilla/controllers/class.discussioncontroller.php(310): Gdn_Pluggable->__call('render', Array)
#11 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(862): DiscussionController->index('2519', 'gpu-memory-ques...', '')
#12 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(279): Gdn_Dispatcher->dispatchController(Object(Gdn_Request), Array)
#13 /var/www/forums-dual-sphysics-org/index.php(29): Gdn_Dispatcher->dispatch()
#14 {main} |
| Notice |
rich is deprecated. Use FormatService::renderHtml($content, Formats\RichFormat::FORMAT_KEY) instead.
#0 [internal function]: gdn_ErrorHandler(16384, 'rich is depreca...', '/var/www/forums...', 950, Array)
#1 /var/www/forums-dual-sphysics-org/library/core/functions.general.php(950): trigger_error('rich is depreca...', 16384)
#2 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1729): deprecated('rich', 'FormatService::...')
#3 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1479): Gdn_Format::rich('[{"insert":"Hi ...')
#4 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(24): Gdn_Format::to('[{"insert":"Hi ...', 'Rich')
#5 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(170): formatBody(Object(stdClass))
#6 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/comments.php(19): writeComment(Object(stdClass), Object(DiscussionController), Object(Gdn_Session), 1)
#7 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/index.php(53): include('/var/www/forums...')
#8 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(778): include('/var/www/forums...')
#9 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(1382): Gdn_Controller->fetchView('', false, false)
#10 /var/www/forums-dual-sphysics-org/library/core/class.pluggable.php(217): Gdn_Controller->xRender()
#11 /var/www/forums-dual-sphysics-org/applications/vanilla/controllers/class.discussioncontroller.php(310): Gdn_Pluggable->__call('render', Array)
#12 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(862): DiscussionController->index('2519', 'gpu-memory-ques...', '')
#13 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(279): Gdn_Dispatcher->dispatchController(Object(Gdn_Request), Array)
#14 /var/www/forums-dual-sphysics-org/index.php(29): Gdn_Dispatcher->dispatch()
#15 {main} |
| Notice |
rich is deprecated. Use FormatService::renderHtml($content, Formats\RichFormat::FORMAT_KEY) instead.
#0 [internal function]: gdn_ErrorHandler(16384, 'rich is depreca...', '/var/www/forums...', 950, Array)
#1 /var/www/forums-dual-sphysics-org/library/core/functions.general.php(950): trigger_error('rich is depreca...', 16384)
#2 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1729): deprecated('rich', 'FormatService::...')
#3 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1479): Gdn_Format::rich('[{"insert":{"me...')
#4 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(24): Gdn_Format::to('[{"insert":{"me...', 'Rich')
#5 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(170): formatBody(Object(stdClass))
#6 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/comments.php(19): writeComment(Object(stdClass), Object(DiscussionController), Object(Gdn_Session), 2)
#7 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/index.php(53): include('/var/www/forums...')
#8 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(778): include('/var/www/forums...')
#9 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(1382): Gdn_Controller->fetchView('', false, false)
#10 /var/www/forums-dual-sphysics-org/library/core/class.pluggable.php(217): Gdn_Controller->xRender()
#11 /var/www/forums-dual-sphysics-org/applications/vanilla/controllers/class.discussioncontroller.php(310): Gdn_Pluggable->__call('render', Array)
#12 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(862): DiscussionController->index('2519', 'gpu-memory-ques...', '')
#13 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(279): Gdn_Dispatcher->dispatchController(Object(Gdn_Request), Array)
#14 /var/www/forums-dual-sphysics-org/index.php(29): Gdn_Dispatcher->dispatch()
#15 {main} |
| Notice |
rich is deprecated. Use FormatService::renderHtml($content, Formats\RichFormat::FORMAT_KEY) instead.
#0 [internal function]: gdn_ErrorHandler(16384, 'rich is depreca...', '/var/www/forums...', 950, Array)
#1 /var/www/forums-dual-sphysics-org/library/core/functions.general.php(950): trigger_error('rich is depreca...', 16384)
#2 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1729): deprecated('rich', 'FormatService::...')
#3 /var/www/forums-dual-sphysics-org/library/core/class.format.php(1479): Gdn_Format::rich('[{"insert":{"me...')
#4 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(24): Gdn_Format::to('[{"insert":{"me...', 'Rich')
#5 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/helper_functions.php(170): formatBody(Object(stdClass))
#6 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/comments.php(19): writeComment(Object(stdClass), Object(DiscussionController), Object(Gdn_Session), 3)
#7 /var/www/forums-dual-sphysics-org/applications/vanilla/views/discussion/index.php(53): include('/var/www/forums...')
#8 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(778): include('/var/www/forums...')
#9 /var/www/forums-dual-sphysics-org/library/core/class.controller.php(1382): Gdn_Controller->fetchView('', false, false)
#10 /var/www/forums-dual-sphysics-org/library/core/class.pluggable.php(217): Gdn_Controller->xRender()
#11 /var/www/forums-dual-sphysics-org/applications/vanilla/controllers/class.discussioncontroller.php(310): Gdn_Pluggable->__call('render', Array)
#12 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(862): DiscussionController->index('2519', 'gpu-memory-ques...', '')
#13 /var/www/forums-dual-sphysics-org/library/core/class.dispatcher.php(279): Gdn_Dispatcher->dispatchController(Object(Gdn_Request), Array)
#14 /var/www/forums-dual-sphysics-org/index.php(29): Gdn_Dispatcher->dispatch()
#15 {main} |
Comments
Hi Felix, actually I have tried several times in Tesla GPU card[80 Gb memory], I find the same issue when the total particles is nearly 180 million, i still don't understand. i ever tried the cpu version and find the same issue when total particles is close to 180 million in one simulation case, i thought maybe there is some limitaions with the released verion, or maybe i missed something. see the attached thread https://forums.dual.sphysics.org/discussion/comment/5612#Comment_5612.
@imartinezestevez and @jmdalonso can help here
@yueguotian I do not think it is the same problem for me as my graphics card has much lower VRAM. So I looked around the web for a bit and found this thread in the old Microsoft tech support:
I was wondering whether somebody else has a workstation with Windows 10 available who could confirm that windows is allocating and blocking at least 20% of the VRAM for single processes. For me, it also seemed to be RAM-usage dependent, i.e. higher RAM-usage led to a lower amount of possible memory allocation. Therefore, I cannot use the full potential of the graphics card due to Windows 10? It might answer my question and would actually be a huge deal for running DualSPH on Windows 10.