depends what _exactly_ you're looking for. Supermicro got themselves on my blacklist. Minisforum is okay if you want a desktop playing dressup, but I don't fully trust them long-term yet.
@furicle @RootWyrm 🇺🇦 To be clear - this project is at the point of "I don't think the CEO has realized we'll need to rent a shelf to store this", so I don't know where it's going to go.
@RootWyrm 🇺🇦 Who is paying: My CEO is asking for numbers, so probably no one - but I would like to collect realistic ones.
Basically - we drank some AI koolaid and then realized it's not cost effective - so they want some numbers on things. So I'm just wondering what vendors people are using these days that aren't the big expensive ones.
I know back in the day Supermicro was the go-to - and Dell is what a lot of people are using as IBM/Lenovo burns its entire brand down. But is there anything that's doing less mainstream work.
so, ballpark, it very much depends on the GPU class. That's your 90% cost, no joke.
ASRock 8U8X-TURIN2 I can't give numbers on, as it's prelim. So, you'd be looking 4U8G-GNR2.
Base chassis, ~$8-10k. CPU, >=$6k. RAM, >=$4k GPU, minimum is going to be NV L40S - MINIMUM. Those are $12k. Each. Minimum. To actually get them expect to pay $20k. Tariffs push that to $19k before 'fuck you' premiums. That's PER GPU and only 48GB which limits the models significantly.
Good choice! AMD has vastly superior FP performance, that's just a statement of fact, and they work great.
MI210 64GB's are $10,000 each with a 12+ week leadtime.
MI300-series are ODM only. Not 'OEM only.' Original DESIGN Manufacturer. Meaning Dell, HPE, Lenovo, etc. A Dell XE9680 will cost over six digits before opex.
sure, if you toss cost considerations out, there's options. Except my point is, there isn't.
I am VERY good at sourcing hardware, thanks to 30+ years as an ODM/OEM. And I could not get you an RTX6000 Blackwell for anything, and the soonest I could get you an H100 is "maybe." This is not factoring in tariffs.
It's just the fact that you are fighting every idiot on Earth for some of the worst yields I have ever seen. And NV is restricting most of the supply to high-profit ODM.
yeah. Honestly the only way you'll get numbers on a single server is Dell/Lenovo/HPE/etc. and it'll be a 9+ month wait unless you have a multi-million per-quarter relationship.
MOQs on ASRockRack and Mitac/Tyan are... prohibitive to say the least. The delusional are simply buying up all the production, continuously. I can get singles but I only do full cab HPC stuff these days.
also bear in mind that as of two days ago, ALL supply from everyone evaporated. Customers are scrambling to get hard commits for every single piece of already onshore hardware for literally everything. And I do mean *everything*.
@silverwizard Kinda a screwball way about it, but a Mac studio cluster maybe? 512GB of unified memory means you can run pretty large models after you cluster a few. 😁
da_667
in reply to silverwizard • • •silverwizard likes this.
silverwizard
in reply to da_667 • • •RootWyrm 🇺🇦
in reply to silverwizard • • •silverwizard
in reply to RootWyrm 🇺🇦 • • •furicle
in reply to silverwizard • • •<joke incoming...>
@silverwizard @rootwyrm
> It needs to be full of GPUs and live in a
> rack and LARP that it's important
like those that admin it?
</hides>
silverwizard
in reply to furicle • • •furicle
in reply to silverwizard • • •silverwizard likes this.
silverwizard
in reply to furicle • • •furicle
in reply to silverwizard • • •RootWyrm 🇺🇦
in reply to silverwizard • • •silverwizard
in reply to RootWyrm 🇺🇦 • • •@RootWyrm 🇺🇦 Who is paying: My CEO is asking for numbers, so probably no one - but I would like to collect realistic ones.
Basically - we drank some AI koolaid and then realized it's not cost effective - so they want some numbers on things. So I'm just wondering what vendors people are using these days that aren't the big expensive ones.
I know back in the day Supermicro was the go-to - and Dell is what a lot of people are using as IBM/Lenovo burns its entire brand down. But is there anything that's doing less mainstream work.
RootWyrm 🇺🇦
in reply to silverwizard • • •so, ballpark, it very much depends on the GPU class. That's your 90% cost, no joke.
ASRock 8U8X-TURIN2 I can't give numbers on, as it's prelim. So, you'd be looking 4U8G-GNR2.
Base chassis, ~$8-10k.
CPU, >=$6k.
RAM, >=$4k
GPU, minimum is going to be NV L40S - MINIMUM.
Those are $12k. Each. Minimum. To actually get them expect to pay $20k. Tariffs push that to $19k before 'fuck you' premiums. That's PER GPU and only 48GB which limits the models significantly.
RootWyrm 🇺🇦
in reply to RootWyrm 🇺🇦 • • •if you want to do current-ish models which have grown out of control, you're immediately into RTX PRO 6000's at "no."
Just no. Hard no. Full stop. You cannot get one.
Which shoves you into the 94GB H100's, which have a 8+ week leadtime and are $31,000. Each.
RootWyrm 🇺🇦
in reply to RootWyrm 🇺🇦 • • •"Ah but what about AMD?"
Good choice! AMD has vastly superior FP performance, that's just a statement of fact, and they work great.
MI210 64GB's are $10,000 each with a 12+ week leadtime.
MI300-series are ODM only. Not 'OEM only.' Original DESIGN Manufacturer. Meaning Dell, HPE, Lenovo, etc.
A Dell XE9680 will cost over six digits before opex.
silverwizard
in reply to RootWyrm 🇺🇦 • • •RootWyrm 🇺🇦
in reply to silverwizard • • •sure, if you toss cost considerations out, there's options.
Except my point is, there isn't.
I am VERY good at sourcing hardware, thanks to 30+ years as an ODM/OEM. And I could not get you an RTX6000 Blackwell for anything, and the soonest I could get you an H100 is "maybe."
This is not factoring in tariffs.
It's just the fact that you are fighting every idiot on Earth for some of the worst yields I have ever seen. And NV is restricting most of the supply to high-profit ODM.
silverwizard
in reply to RootWyrm 🇺🇦 • • •RootWyrm 🇺🇦
in reply to silverwizard • • •yeah. Honestly the only way you'll get numbers on a single server is Dell/Lenovo/HPE/etc. and it'll be a 9+ month wait unless you have a multi-million per-quarter relationship.
MOQs on ASRockRack and Mitac/Tyan are... prohibitive to say the least. The delusional are simply buying up all the production, continuously. I can get singles but I only do full cab HPC stuff these days.
silverwizard likes this.
RootWyrm 🇺🇦
in reply to RootWyrm 🇺🇦 • • •And I do mean *everything*.
silverwizard likes this.
JB Carroll
in reply to silverwizard • •silverwizard
in reply to JB Carroll • • •Ben Zanin
in reply to silverwizard • • •silverwizard likes this.
silverwizard
in reply to Ben Zanin • • •