I’ve got a very special piece of hardware coming my way for review: a Blackbird Secure Desktop from Raptor Computing Systems. The Blackbird is a desktop PC with an IBM POWER9 processor that is open source from top to its very bottom – no firmware blobs, no management engines, no proprietary BIOS.
As the product page details:
The Blackbird™ mainboard is an affordable, owner-controllable, desktop and entry server level mainboard. Built around the IBM POWER9 processor, and leveraging Linux and OpenPOWER™ technology, Blackbird™ allows you to secure your data without sacrificing performance. Designed with a fully owner-controlled CPU domain, you can audit and modify any portion of the open source firmware on the Blackbird™ mainboard, all the way down to the CPU microcode. This is an unprecedented level of access for any modern desktop-class machine, and one that is increasingly needed to assure safety and compliance with new regulations, such as the EU’s GDPR.
I don’t yet know what exact specifications my review unit will have, but I’m assuming it’ll be the base model that has the 4-core POWER9 processor with SMT4 (4-way multithreading). I do know it’ll come with an AMD Radeon Pro WX4100 LP, which will be the only piece of hardware requiring card-side proprietary firmware (but it’s optional, since the mainboard itself has basic open source graphics capability too).
I don’t usually do this, but there’s a first thing for everything, so here we go: do any of you have any questions about this exotic hardware you want me to try and answer? Specific things to look into? I’ll also be able to ask some questions to Raptor’s CTO, so there’s a lot of opportunity to get some serious answers.
I’ll try to take as many suggestions into account as I can. The current estimated delivery date is 6 August, so expect the actual review in late August or early September. Also I’m sorry for the title pun.
I would be very interested to know how well OpenBSD and FreeBSD support works.
I would be interested in this as well. I don’t have high hopes, but it would be interesting to know what’s broken.
I can confirm FreeBSD 12.1 runs perfectly under QEMU/KVM. I have not tested installing it as host OS though.
Cool. Thanks!
I would mostly be interested in it as the host.
OpenBSD’s first powerc64 commit only happened 2 months ago. It’s only the bare minimum kernel and some userland bits to get things going. A fairly good portion of the ports tree is building and a lot more will come with a bit more work. Give kettenis a few more months and the hardware support will improve quite a bit. OpenBSD has drm from the 5.7 kernel. X is building but AFAIK not actually running yet, but I’m sure that and amdgpu is not that far off.
As obrando mentions, current POWER9 support for OpenBSD is quite rudimentary; FreeBSD is much better supported I think.
If you are curious, I would keep an eye on the RCS Wiki’s OS list:
https://wiki.raptorcs.com/wiki/Operating_System_Compatibility_List
FreeBSD has a page mentioning what is being worked on here:
https://wiki.freebsd.org/powerpc/POWER9
For OpenBSD status, you may need to peruse their mailinglist archives, the equivalent page has limited information at the moment:
https://www.openbsd.org/powerpc64.html
One of the best workstation I have ever purchased. It runs Fedora 32 very reliably. What I like about it is the superb support for driver, for example amdgpu driver works perfectly for my Radeon Vega 64 card. I could play games, watch movie and do work like I do on my other x86 laptop.
I highly recommend potential buyers to go for more memory, as much as possible if you want to have more concurrency when compiling application (some does consumes lots of memory).
I’ve had good experiences with a PPC NAS, but it was so long ago.
I’m always interested in benchmarks, cpu/video/memory. I am also curious how well vitalization works on architectures outside of x86. If you have the ability to measure power usage, idle and under load, that’s sometimes useful for comparison.
Most people don’t think about it right away, but one of the first things I test on a new system is lmsensors and other system diagnostic & optmization software since that determines how well you’ll be able to keep an eye on the hardware. My current (gigabyte) motherboard isn’t supported by linux and I cannot monitor the fans or temps, which is so annoying. The sensors are there, but I can’t read them while the system is running, only in the bios. Gigabyte hasn’t contributed drivers to linux.
Phoronix has done a CPU benchmark against Blackbird and Raptor system a while a go. GPU is on-par with Linux x86_64, in my case I benchmark my Vega64 card.
Virtualization works great, I’ve been running multiple VMs with QEMU+KVM to host a Kubernetes cluster for learning purpose.
I have not done any measurement on power usage so I could not comment on that. And regarding sensor stats, the support is perfect, lmsensors reports them correctly.
I would like to know if AROS can be made to run on it.
Congratulations, it looks like a nice machine.
That being said, whenever I get excited about an exotic device, I am put down by the price tag. The base model is listed for over $3,500, and even simple upgrades like 16GB RAM cost a lot. I know they need to make a profit to stay afloat, but I cannot get myself to expense such an amount.
If I would really want to spend at that level, I can go with a standard workstation from SuperMicro for example:
https://www.thinkmate.com/systems/supermicro/superworkstation
At the same price the feature set is not even comparable.
Tips: Buy the refuribished RAM from eBay, I got my 64GB DDR4 ECC ( 2 x 32GB) for just $150 USD.
The article has pointed out earlier, the key difference is openess. And have I mentioned that the blackbird is eerily quiet compared to other workstation?
Well, I am still adjusting to the $40 I paid for the Rasperry PI (was it a spur of the moment purchase, yes, yes it was). I like the open machines, but not being mainstream comes with a very high price tag.
As for the silence, it is possible to run Xeons with no fans. It takes some effort, but there are several ways to do that. one example: https://www.youtube.com/watch?v=DiLOsGTw78w . SuperMicro also offers silent versions of their PSUs, or you can try to replace it with an aftermarket one. Overall getting a silent and powerful workstation is possible.
Each to their own. The Blackbird is not for everyone but I do not think the price tag is high. People used to pay much more for a PowerMac G5.
The current Mac Pro starts at $6k. $3.5k isn’t unusual for enterprise hardware.
The123king
It depends on if the computer is intended to be a workstation or a server. The mac pro isn’t a bad price for an enterprise server, but it is overpriced and underspeced as a $6k workstation. You’re actually looking at $10k+ delivered once you upgrade to less bad specs.
With this blackbird system, I wouldn’t say it’s unreasonable for a workstation with some enterprise features, but it’s definitely quite expensive for the consumer space. As speced it’s kind of minimalist (128GB NVMe, 8GB ram, 300W power supply, 4c CPU). Enterprise components can explain the cost, but for most consumers a commodity x86 system would go a lot further for their needs.
For a workstation, I personally would stay away from microatx and opt instead for a full tower with a better power supply and upgradability. While Rapter sells products that fit the bill, the price tag is pretty high and I kind of wish they had 10gbp ethernet.
https://www.raptorcs.com/content/base/products.html
I buy servers used at significant discounts. Real enterprise hardware depreciates in value quickly, but is actually very durable. I won’t link it because I don’t want to endorse it, but I looked up “power8 server” (a generation behind) on ebay and this refurbished system is $3000.
Assuming secondhand hardware is suitable, you can buy used stuff that costed a fortune a few years ago. Obviously it always comes down to what one can afford as well as pecking order. I’ve seen executives pimped out with ludicrously expensive machines even though it’s wasted on them “just because they can”. Meanwhile normal employees don’t get much priority, haha.
This is for people or organizations that require a completely free/open software and hardware stack to enforce their privacy and security.
SuperMicro offers only standard hardware, comes with IME/PSP as standard (for example) and cannot be deactivated: you are forced to load a closed binary driver that moreover operates at a lower level than OS, silently and undetectable.
How hard is it to install different distros, like CentOS, Fedora, or Ubuntu?
How big is it? I can’t tell if it’s a full-height case or half-height.
How many disks can you stuff in the thing? It looks like 2x NVME plus 6x SATA is supported by the board, but does the case support all of those drives?
What does the upgrade situation look like? OEMs are notorious for only including PSU attachments, drive trays, and cables for what the machine was ordered with. I understand being stingy with SATA cables, but PSU connectors and drive trays are different.
How is it as a headless server? Does the IP-KVM functionality of OpenBMC work pretty well, and how well does it support virtual media for installation?
> How hard is it to install different distros, like CentOS, Fedora, or Ubuntu?
Very straight-forward. I could install Fedora 32 and Ubuntu 20.04
> How big is it? I can’t tell if it’s a full-height case or half-height.
The motherboard itself is a Mini-ITX, you could fit that into any computer case
> How many disks can you stuff in the thing? It looks like 2x NVME plus 6x SATA is supported by the board, but does the case support all of those drives?
There is ZERO M2 slot for NVME. You could use PCI-E 8x to M2 adapter card for that job. It does come with 4 SATA internal ports. I could not comment on the stock computer case that raptor computing is selling.
> What does the upgrade situation look like? OEMs are notorious for only including PSU attachments, drive trays, and cables for what the machine was ordered with. I understand being stingy with SATA cables, but PSU connectors and drive trays are different.
I could not comment on this but I do not see any reason why the PSU does not come with the same amount of cables like that of x86_64 computer. Please note that the PSU is just a standard ATX, there is nothing proprietary here.
> How is it as a headless server? Does the IP-KVM functionality of OpenBMC work pretty well, and how well does it support virtual media for installation?
:shrug: I am unsure
According to the website, it’s micro-ATX, not mini-ITX.
Yes, my mistake. It is Micro ATX.
Cool, thanks!
I was asking about the case and how it handles the PCIe slots specifically. Half-height vertical or full-height horizontal via riser cards.
Yeah, the number of SATA ports is a mistake on my part. I misread the description. “4 6 Gb/s SATA ports….”
Then how can the system be upgraded to 2x NVMe drives? The current configurator lets me add 2x NVMe drives.
Are they using U.2 drives rather then M.2?
I have some Dell minitowers which should be able to power a GPU, but I’m limited to GPUs which pull <75W from the PCIe slot because the PSU doesn't have a connection for it. I also have to use SATA power splitters to get 4x SATA drives because the PSU only has 2x SATA power connections.
Things like that.
As has already been said, and you quoted, the only option to integrate NVME with this board is through a PCI-Express adapter card, for example in the x8 slot. You can get these for 10 bucks and you plug & screw the drives directly onto the adapter. It works the same as an NVME m2 connector on a board, which, essentially, is a PCI-Express slot in a different format.
That tells me nothing of the official solution. That tells me someone bought an NVMe adapter, and stuck it in a PCIe slot, which I already know about and use regularly, so you’re not telling me anything I don’t already know.
This also doesn’t tell me if the board firmware supports PCIe bifurcation or if the NVMe adapter needs to have a bridge chip to mux the signals.
Next, if you look at the configurator for the actual board instead of the preassembled desktop, you’ll see an option for an OCulink PCIe expander and OCulink cables. Are any of these these being leveraged to provide NVMe in the completed system? They don’t say.
Then there are the ports which look like Mini-SAS ports on the board, which aren’t explained. What are those ports? There are lots of things about the the boards which aren’t explained very well.
Now if an NVMe adapter is the official solution, this is annoying. The board only has an x16 and an x8 PCIe slot. In a normal desktop x16 slot would get taken up by a GPU, and the x8 slot would get taken up the NVMe adapter. This would make me pick between having NVMe storage or a NIC faster then 1Gbps.
@Flatland_Spider To quote the Blackbird User’s Guide “PCIe slots cannot be split without an external PCIe switch card”.
It may help to understand that there is no “chipset” with POWER9. Each PCIe slot runs directly into the CPU, and on these Nimbus-Sforza modules, you have 48 PCIe lanes that can be connected to a maximum of 6 endpoints. If you look at the block diagram on the RCS Wiki page, you can see that these are being used to connect to:
x1 BMC (AST2500)
x2 NIC (BCM5719)
x1 USB Controller (TUSB7430)
x8 SATA Controller (88SE9235)
x8 Slot 1
x16 Slot 2
On the Talos II boards, an optional SAS controller with proprietary firmware takes the place of the SATA Controller, and on the non-Lite board 3 additional slots (x16, x16, and x8) and a μPCIe port are available when using a second CPU.
References:
RCS Wiki Blackbird page: https://wiki.raptorcs.com/wiki/Blackbird
Blackbird (C1P9S01) User’s Guide: https://wiki.raptorcs.com/wiki/File:C1P9S01_users_guide_version_1_0.pdf
page 24 of POWER9 Sforza Datasheet version 1.8: https://wiki.raptorcs.com/wiki/File:POWER9_Sforza_DS_v18_13JUN2019_pub.pdf
RCS Wiki Talos II page: https://wiki.raptorcs.com/wiki/Talos_II
> How is it as a headless server? Does the IP-KVM functionality of OpenBMC work pretty well, and how well does it support virtual > media for installation?
The IP-KVM functionality of OpenBMC works very well. When you connect to the BMC, you can access a serial console or a graphical console. There is an option to provide virtual media, but I’ve never tried it. If that doesn’t work (though I imagine it does), petitboot can be configured to easily netboot an installer.
> How many disks can you stuff in the thing? It looks like 2x NVME plus 6x SATA is supported by the board, but does the case support all of those drives?
I have 4x 2.5″ SSDs and 1 NVMe in my system. The SuperMicro NVMe adapter I used can hold up to 4. Just buy a bare board, CPU and heatsink and then provide your own case. I use a SuperMicro case, but almost any case will work with it as long as it’s reasonably well-built and standards compliant.
Good to know, thanks!
I miss read the SATA specs.
Does the BIOS (Is it a BIOS/UEFI, or something like OpenFirmware?) support PCIe bifurcation, or does it need a NVMe adapter with a bridge chip on it?
I’m trying to get away from building my own desktops. The hassle isn’t worth it.
> Does the BIOS (Is it a BIOS/UEFI, or something like OpenFirmware?) support PCIe bifurcation, or does it need a NVMe adapter with a bridge chip on it?
There is no bifurcation on the motherboard.
You do not need NVME M2 adapter with bridge chip if you want to use one M2 SSD.
If you want to run more than one, yes you would need to buy the adapter card with bridge chip.
That’s disappointing. 🙁 The multiple NVMe adapters without the bridge chip are easier to find and cheaper.
Yeah, that’s what I’ve doing with older equipment. The single adapters work well for the i5 SFF desktops I have since they need low profile cards.
I would like to add some 4x NVMe adapters to some of my “servers”, but the cost of the adapter with the bridge chip and drives is more then what I paid for the “server”. XD
What’s the main use case for such a machine?
How much “bang for buck” do you get? Is it quite powerful?
Is there other products in the pipeline using a similar architecture? Will there be cheaper models?
Would it run AmigaOS?
Security. People seem to forget that all sorts of vulnerabilities disappear if you avoid running non-free software. For instance none of the Intel CPU vulnerabilities affect people that are not running any untrusted code (not even inside sandboxes such as the JavaScript sandbox in your browser).
Obviously this limits functionality compared to typical environments. Having the ability to procure systems for which you don’t have to rely on a specific third party for any security aspects is attractive for specific use cases. RYF certified hardware is the only hardware available that’s suits this.
Will there be small models as well. Something like the MacMini or a NUC with an internal PSU would be nice. I would say something like an RPI SBC would be cool, but that might be too much.
For me I use it like a normal x86_64 workstation. That is programming, personal banking, movies, music, games and anything in between. Please think of it like a high end workstation that runs Linux. 99% Linux apps are compatible on this platform if you run ppc64 LE variant of the Linux kernel.
It is a beefy machine (depending on how many cores you go for), you could expect high efficiency if your app takes full advantage of those cores.
I am unsure if it could run AmigaOS.
I would love to have one if i could run AIX on it.
There may be some difficulty with that, AIX and IBM i are meant to run under the proprietary PowerVM hypervisor. My understanding is that POWER9 machines running PowerVM may be using different firmware as well, rather than the open-source OpenPOWER firmware, which is sometimes referred to as PowerNV (Not/Non-Virtualised).
I’ve been using an 8 core SMT4 Blackbird full-time since roughly this time last year. It’s been entirely pain-free under Void Linux – https://voidlinux-ppc.org/ . I can’t recommend the hardware and Void enough!
I purchased the motherboard, CPU and heatsink from Raptor. RAM, GPU, and drives I purchased on my own / had available. It’s not bottom-of-the-barrel pricing, but you do get an owner-controlled workstation that is very performant.
There’s a few things I want to see.
1. Web benchmarks in Firefox with the usual suite of security addons installed (uBlock Origin, PrivacyBadger, etc.)
2. Xonotic benchmarks as a proxy for Free Software GPU loads.
3. Multithreaded scientific computing benchmarks.
4. Related, scientific computing software -availability-. I want to see how much of, say, R, Julia, and NumPy/SciPy/Pandas works.
5. Give the OpenBMC hardware a workout. I have an old scavenged Supermicro rackmount box I’m looking at replacing soon, and a flaky BMC is why.
> 1. Web benchmarks in Firefox with the usual suite of security addons installed (uBlock Origin, PrivacyBadger, etc.)
Pretty much exactly the same. No differences as far as I could see.
> 2. Xonotic benchmarks as a proxy for Free Software GPU loads.
Good suggestion. I am going to try big benchmark mode on this box.
> 3. Multithreaded scientific computing benchmarks.
Any suggestion?
> 4. Related, scientific computing software -availability-. I want to see how much of, say, R, Julia, and NumPy/SciPy/Pandas works.
Python, Erlang, Elixir, Ruby, Golang, Java, Ruby I could confirm working out of the box. I am unsure about R or Julia. Feel free to suggest any workload that you want to seem, I could run and report back to you
> 5. Give the OpenBMC hardware a workout. I have an old scavenged Supermicro rackmount box I’m looking at replacing soon, and a flaky BMC is why.
What’s workout here?
3,4: https://github.com/jonathanBieler/ScientificComputingBenchmarks.jl
5. Render the system inaccessible from its primary network interface and use the BMC to turn it back on. Do an OS version upgrade. Get into GRUB and boot the kernel with special parameters.
I’ve been about to pull the trigger on one of these since the end of 2018, but I keep stopping at the last moment due the scale of the investment and looming questions about/risks of important functionality not being available.
I understand there is now a Chromium port that works, but that V8 is highly optimized for x86 and ARM and reportedly runs like ass on POWER. How well (or poorly) does Chromium actually perform on POWER9 for JS-heavy sites like YouTube, Maps, GMail, etc.? Firefox is mature and maintained on the platform, and I moved back to Firefox in May after ~11 years away, but I find there are still some sites that fail to render, or limit functionality [YouTube!], or even outright refuse to run, unless you spoof the user agent.
I would like to know how well binfmt_misc works for running x86-only code on POWER: both performance and level of breakage. Are you able to do anything silly with Wine + binfmt_misc to run Windows applications “directly?” (This will also be useful once production ARM Macs are in the wild.) Reference: https://forum.winehq.org/viewtopic.php?t=17701
I’m also curious how usable Windows 10 is under QEMU on POWER. I imagine not very, but since $work requires me to use a Citrix client, and is now blocking Linux endpoints entirely, I’ll have to make do. Yes, of course i could “just get another computer to run Citrix”– that is very much not the point.
The last benchmarks I looked at for the Talos II/Blackbird gave me the impression that while the platform has potential, the potential is not being realized due to limited optimization, and right now POWER9 is easily beaten in most normal compute tasks by an AMD64 system costing a quarter as much. While the Raptor platform offers many other advantages, it seems current performance is not one of them, and I wonder if it ever will be during the POWER9’s useful lifetime.