It’s that time of year again – time to tinker with my deep learning rig, Eve! (Yes, like Wall-E!)
As an experienced ML engineer, I typically develop and test my training scripts on Eve before scaling up to cloud instances in AWS or GCP. My current setup is a solid multi-GPU machine with 2 x 3090 cards, each boasting 24GB of VRAM.
However, with increased training and model evaluation demands, I noticed Eve’s power consumption had climbed significantly over the past year (thanks to the insightful data logging from my Home Assistant instance running on a Raspberry Pi). This prompted me to explore an upgrade focusing more on efficiency. Enter the 4060Ti!
On paper, the 4060Ti might seem memory bandwidth-bottlenecked compared to the 3090, but its improved architecture requires significantly less power. While it might be slightly slower for training, the energy savings are compelling. Plus, I snagged a great deal on the 4060Ti with its 16GB of VRAM.
The current GPU market, particularly for Nvidia cards, is still quite inflated. A pre-owned 3090 goes for around $1800 CAD – almost three times the price of a new 4060Ti! While AMD offers more competitive pricing, my work relies heavily on CUDA programming and the ML ecosystem remains tightly integrated with it. Based on my recent experiments (detailed in a separate post [link]), I believe it will take some time for AMD’s RoCM to catch up with Nvidia’s for AI applications.
My plan is to make the 4060Ti my primary driver GPU and utilize the two 3090s for heavy workloads. The upgrade will happen later this week once my AM5 chipset motherboard arrives – I’m incredibly excited about this build! Stay tuned for a follow-up post detailing Eve’s revamped performance, including power consumption and ML benchmark stats like token generation/sec for LLM workloads and other relevant metrics.
Stay tuned!
