Nvidia DGX Spark Rack & Cooling

After about seven months, one of my DGX Sparks started crashing under heavy load. It typically crashed after about two hours of sustained load (e.g., when running DeepSeek V4 Flash). At first, I thought it was an out-of-memory (OOM) error. I’d seen that error a couple of times before when I was running tools on the same host I used for inference. Occasionally, I’d forget to disable a cron job that ran a threat intelligence processing pipeline. When the job kicked off during inference, the host would throw an OOM error. Whenever one of these crashes occurred, I’d have an agent debug it. I keep a working directory specifically for debugging these kinds of issues. When a new issue comes up that I’d like an agent to debug, I create a folder with a descriptive name, cd into it, and start an agent. The benefit of this approach is that if the issue comes up again, I can resume the session and pick up where I left off. I’ve found that this saves a lot of time on recurring issues, especially networking problems.

After a couple of sessions trying to track down the cause of these crashes with an agent, I decided to check the NVIDIA forums. If you haven’t visited them, they’re a gold mine of tips and ideas. One tip was to run NVIDIA’s DGX Spark Field Diagnostics tool to check for hardware failures. Sure enough, the tool identified a hardware issue. Anyone who frequents the NVIDIA forums has probably heard about these failures. Overheating is a familiar complaint among DGX Spark owners. Luckily (well, kind of), I bought a three-year warranty when I got mine from Micro Center. Their warranty is a good deal, but there are some gotchas. If the machine is still under the manufacturer’s warranty, Micro Center can replace it. That’s a one-and-done deal unless you buy another warranty. The second gotcha is that if the machine fails after the manufacturer’s warranty expires but within Micro Center’s three-year coverage period, Micro Center gives you store credit for the original purchase price.

That’s a great deal in a normal market, but Spark prices have doubled since I bought mine. If it has another thermal failure after the one-year mark, the store credit will only cover about half the cost of a replacement at today’s prices. If anyone asked me whether they should get a Spark, I’d say no. At this point, it feels like a Spark’s shelf life is about as long as the manufacturer’s warranty. After that, it’s a crapshoot. 

To give my Spark a better shot at staying cool, I started looking into cooling systems. Unfortunately, most of the setups I found for DGX Sparks required either a 19 inch server rack or a 3D printer, and I don’t have either. I remembered seeing a post and a video by Jeff Geerling about micro racks, particularly the DeskPi. I couldn’t find any photos of anyone using a DeskPi to cool DGX Sparks, so I decided to give it a try.

For this build I used the following: 

  • GeeekPi 8U Network Rack, 10 inch Mini Server Rack for Network, Servers, Audio, and Video Equipment, DeskPi RackMate T1, 7.87 inch Depth
  • 2 ELUTENG Dual USB Computer Cooling Fan High Speed 120mm Fan Adjustable 5V USB Port Heatsink Cooling Fan 12cm
  • Wire 
  • Electrical Tape 
Here are photos describing the setup. Unfortunately, I didn’t take any photos during the build process. 
 
The first task is to connect the two fans (assuming the rack has already been put together). The corners of the fans have black plastic covers over the screws. These covers that are used to prevent any rattling sounds and act as padding. Place the two fans next to each other and take off the two covers off that touch each other in the middle. Then use some wire to tightly connect the two screws and then wrap with electrical tape. The connection will look like the below photo. 


Below is an example of the black plastic washer previously described.  The screws of the fan do not match the width of the GeekPi but the metal beam (provided by GeekPi) can be to clamp down the fan. 


Here is a photo of the washer clamped down with another beam at the bottom. 


The following photo is a view of the fan from the top. The fan is blowing air into the rack. 



Here is another view of the fans straight on. The fans are connected to the rack in the four corners. The two black cross beams at the top and bottom of the fans keep them mounted. While this seems like it would not be a solid way to mount the fans it works very well. There is no rattling and the fans do not move at all. 




Here is a side shot of the front where the fan is being blown out. The cross beam above the DGX Spark fits almost perfectly. I added a little bit of electrical tape to the bottom of the beam so it doesn’t scratch them when taking them in and out. It should be noted that the rack at the top was an additional purchase. The default rack mentioned at the top only comes with one rack.

Here is the depth of the rack and the DGX Spark. There is plenty of space for the fans to push air into the DGX,  I included measurements for others who might want to understand the depth of the rack and the DGX, 

Here is the height of the rack with the DGX Sparks. 






Another example front view. The switches for the fans can be seen coming into the front in the bottom section.



Close up photo of the DGX Sparks, fans and the fans switches. 



Hopefully this helps others with a setup for their Nvidia DGX Sparks. I have had this setup for about two weeks and I like it. I can easily move it around, it fits underneath my desk and the Sparks keep my feet warm. Feel free to leave a comment or send me any email (Alexander dot Hanel at gmail dot com) if you have any questions. Cheers. 

No comments:

Post a Comment