The Rough Guide to MBus Modules
Introduction Buses Modules Systems Chips Miscellany

General Module Configuration Rules

This page covers the general rules for CPU- and module-configurations. There are also system-specific limitations, which are described elsewhere.

The rules are expressed in order of priority, ie: if a particular configuration satisfies an individual rule, it must still be checked against all the earlier rules, too.

NOTE: mixing modules of different types, revisions, speeds, etc. is not supported by any of the system vendors, even in those cases where it does work.

And remember, not all modules of a given "type" are the same. For instance, Sun parts 501-2607 (SM51) and 501-2617 (also SM51) are not compatible with each other in some systems. Not all SM51s are the same, and that goes for many module "types".

  1. The individual system-boards of the SPARCcluster-1/2/3/4 should be treated as independant systems for the purposes of CPU configuration and all the remaining rules.

  2. The active system firmware (PROM) must be able to recognise all the modules present.

    See the individual system descriptions for details.

    Be especially careful with the SPARCserver-1000, SPARCserver-2000 and Cray CS6400 which have a dynamically-selectable active PROM.

    In some systems, double-CPU modules may instead be "recognised" and work as single-CPU modules, possibly depending on PROM and system board revisions.

  3. You cannot mix Cypress, SuperSPARC or HyperSPARC modules with each other.

  4. You cannot use Cypress, HyperSPARC or uncached SuperSPARC modules on XBus systems.

  5. To use uncached modules, the MBus must be clocked at the module's rated speed or slower. The CPU is clocked at the same speed as the MBus.

    eg: SM30 will only work on an MBus running no faster than 36MHz.

    eg: SM50 runs at 40MHz in the SPARCstation-10 (40MHz MBus), but at 50MHz in the SPARCstation-20 (50MHz MBus).

    Some systems will automatically adjust their MBus speed to match the module, within limits. For instance the SPARCstation-20 can automatically switch between 40MHz and 50MHz MBus speed (for SM40/41/51/52X versus SM50/61/71/81 modules), but it will not go lower than 40MHz (thus preventing use of SM20/21/30 and the 33MHz flavour SM41).

    Other systems cannot automatically adjust their MBus clock, but must be set explicitly by motherboard jumpers or firmware settings.

    Unfortunately, some systems have an MBus clock fixed in stone :-(.

  6. On MBus systems, cached SuperSPARC modules must run faster than the MBus, due to a design quirk of the MXCC cache-controller.

    eg: the SM51 forces the SPARCstation-20 MBus down to 40MHz, whilst running itself at 50MHz. Contrast this with the (uncached) SM50, which would let the MBus run at 50MHz.

  7. On XBus systems, modules must run at the same speed as the XBus, or faster.

  8. You cannot mix cached and uncached modules on a single MBus.

    eg: a mixture of SM50 and SM51 will not work.

  9. You may or may not be able to mix modules with different L2 cache sizes.

    For instance, SM61 with SM61-2, or HM125S-512 with HM125S-1024.

  10. The success of mixing a dual-CPU module with a single-CPU module depends on both the system firmware and the operating-system.

    See the individual system descriptions.

    As long as the PROM is happy, Solaris 2.2 can handle three CPUs per MBus/XBus. Solaris 2.1 and SunOS 4.x are an unknown quantity.

  11. For SuperSPARC modules with 2Mb of cache, only dual-XDBus (SPARCcenter-2000) or quad-XDBus (Cray CS6400) systems can utilise the whole 2Mb.

    Other systems will only make use of the first 1Mb of the cache. In fact, some early revisions of the SPARCcenter-2000 also cannot see the 2nd Mb of cache.

  12. You should not mix MXCC cache-controllers of differing major revisions, on the same MBus/XBus.

    For instance, mixing 501-2756 (SM51, MXCC v3.3) with 501-2609 (also SM51, but MXCC v2.x) in the SPARCserver-690 is highly unreliable. Expect crashes, hard-hangs, missed peripheral interrupts, data corruption, and so on.

    But note that 501-2756 (MXCC v3.3) works perfectly together with 501-2780 (MXCC v3.1)... the cache-controllers are different minor revisions, but have the same major revision, which is what matters.

    Also, a pair of modules with v2.x MXCC, or a pair with v3.x MXCC, work fine in the SPARCserver-600 series. Just don't mix them...

    Although you may be able to make some "mixed-MXCC" configurations appear to work, it is asking for trouble. Make sure you have some good backups.

    This counts double for v4.x MXCC cache-controllers, which have extra capabilities over their earlier counterparts. I'm not sure what Solaris would make of a v3.x/v4.x mixture, but don't be surprised if it tries to enable "multi-command mode" on the MXCC that cannot do it, and then crash. SunOS 4.x definately chokes on such an MXCC mixture.

  13. You can mix cached SuperSPARC modules of different speeds (eg: SM51 and SM61), if you are using Solaris 2.5.1 or a later O/S, but you may not be free to choose which module goes in which slot.

    The slot restriction only applies on those systems that can automatically downclock the MBus. For instance, when mixing an SM51 with an SM61 (other than 501-2571)in the SPARCstation-20 (autoselected 40/50 MHz MBus), the SM51 must go in slot 0, because only slot 0 is used in the bus-speed-selection logic. With the SM51 in slot 0, the MBus is downclocked to 40 MHz which suits both modules, but if the SM61 is in slot 0, the MBus runs at 50 MHz which is not suitable for the SM51: the operating-system will crash when the second CPU is brought online. Note that in such a situation the system may well pass all PROM diagnostic tests on both CPUs - the "wrong bus speed" only manifests itself when multiple MBus nodes (cache-controllers, main memory-controller, or the MBus/SBus interface chip) request access to the bus at the same time.

    Older operating systems/releases are an unknown quantity.

    Each module is independantly clocked - the faster module will tend to do more work, because it is faster.

    In MBus systems, the operating system will not especially favour the faster CPU, but will try to distribute work on a "what's available" basis, ie: if both CPUs are idle when a new job comes along, it may be started on the slower CPU.

    So on an MBus, the faster CPU will in general do more work, because it will tend to finish things more quickly and thus be "available" more often.

    However, on XBus systems, it appears that Solaris favours the CPU in slot A over the one in slot B. I can not see any reason why it should do so, but measurements of a symetrical dual-threaded CPU-bound workload on "SM51 with SM61" in both slot-configurations quite clearly show substantially greater system throughput with the SM61 in slot A.

    Simplified summary: for mixed-speed module configurations, the slower module usually must be in slot 0 (slot A) on the MBus, whereas it should be in slot B on the XBus.

    In these "uneven" configurations, utilities such as mpstat, psrinfo and vmstat will report accurately (at least they do under Solaris 2.5.1 and later releases). However, third-party performance-reporting tools (such as the SE Toolkit) may be fooled.

  14. You can mix SuperSPARC-I and SuperSPARC-II CPUs eg: SM51 and SM71. But watch out for those MXCC (cache-controller) versions! See above.

  15. Uncached modules with early-revision SuperSPARC-I CPUs can not be used in multiprocessor configurations.

    That means SM20, SM30, and early revisions of the SM40 (Sun part numbers 501-2219 and 501-2295).

    Solaris 2.2 and later releases will deliberately run in uniprocessor mode in the presence of multiple modules one of which is an early-revision SM40, whereas Solaris 2.1 and SunOS 4.1.3 will crash horribly. It is unknown whether SunOS 4.1.4 will run in uniprocessor mode, or crash.

    The multiprocessor-suitable SM40s are Sun part-numbers 501-2358 and 501-2570.

  16. The only mixed-module configuration officially supported by a system manufacturer is SM520 with SM521 in the SPARCserver-600 series.

    However, if your systemboard/motherboard has opposed MBus/XBus slots, and at least one of SM520 or SM521 works on it, then it can be paired with the "other" one.

    In other words, if SM520 works in the system, and SM521 physically fits, then you can use both together. And vice versa.

  17. You should be able to mix different minor revisions of the same module version. For instance, mixing 501-2708-01 and 501-2708-02 should be just fine.

    However, there are a few cases where this is not true, mainly on the SPARCserver-1000, SPARCserver-2000 and Cray CS6400.

Introduction Buses Modules Systems Chips Miscellany
Mike Spooner, revised 6th December 2000