On server management interface

Alexandru Grigore
IT-CF-FPP

HEPIX 2017 @ KEK, Tsukuba

Agenda


Introduction to BMC, IPMI

CERN BMC firmware upgrade campaign

Redfish API

OpenBMC

Baseboard Management Controller

BMC

Features

Server control and monitoring

Power commands, sensor readings

System events, iKVM + virtual devices

SOL, Boot device control

Fan Speed control, FRU information

PET alerts, PEF, User profiles

Watchdog timers, OEM specifics via raw commands

IPMI


Specification for accessing the BMC (also independently from the OS level)
  • architecture
  • commands
  • event formats
  • data records
  • capabilities used across IPMI-based systems and peripheral chassis

Multiple implementations of IPMI specification


DRAC (Dell Remote Access Card)

HP iLO (Integrated Lights Out)

IBM IMM (Integrated Mangement Module)

SUN ILOM (Integrated Lights Out Manager)

The IPMI specification


It is very useful, but...

provides limited information on the system

implementations come with custom OEM extensions

security was largely ignored

cipher0

ipmitool -I lanplus -C 0 -H 10.9.68.30 -U admin -P any_password chassis power off

null authentication

ipmitool -I lanplus -H 10.9.68.30 -U "" -P "" user set password 2 pwned

passwords stored in clear text

...continuing the non-exhaustive list

SSLv2, SSLv3

no support for TLS1.2

MD5, RC4

Self-Signed certificates pre-installed

Mitigation

Management traffic separation


separate management traffic from data traffic

CERN BMC firmware update campaign

Started: August 2015

With: Intel, Supermicro, Quanta

Found bugs in the firmware and update tools

Reached all servers in CERN datacenter

Completed: July 2017

What we asked for


  • Disable SSL support (all versions)
  • Add support for TLSv1.2 using a 256 bit encryption key
  • Disable all TLS ciphers using less than 128 bits of encryption
  • Support secure TLS session renegotiation only
  • Make TLSv1.2 256 bit ciphers the default ones
  • Increase RSA key strength to at least 2048
  • Possibility to upload CERN CA SSL certificate

What we achieved

SSLScan before patching campaign
SSLScan after patching campaign

Kudos


Manufacturers

Intel, Supermicro and Quanta

CERN suppliers

CERN colleagues

Security: Stefan Luders, Liviu Valsan

Sysadmins: Luca Gardi, Aurelien Gounon

Introducing Redfish API


Scalable Platforms Management Forum (SPMF)

Promoters: Broadcom, Dell, Emerson, HP, Intel, Lenovo, Microsoft, Supermicro, VMWare

Purpose: to create a RESTful interface for simple, modern and secure server management

Redfish API

Introducing Redfish API (contd.)


The RESTful interface allows consumers to interact with resources using a restricted set of verbs


Resources uniquely identified by Uniform Resource Identifiers (URI)


Access is done via HTTPS, disabling HTTPS will disable Redfish API access


URIs examples


"ClearSEL" : "/redfish/v1/Managers/1/LogServices/Log1/Actions/LogService.Reset"

"BMCReset" : "/redfish/v1/Managers/1/Actions/Manager.Reset"

"SystemReset" : "/redfish/v1/Systems/1/Actions/ComputerSystem.Reset"


URI mockups

Read chassis information



import requests
import json

# authenticate 
url = 'https://redfish-004-ipmi/redfish/v1/SessionService/Sessions/'
payload = {"UserName":"******","Password":"******"}
headers = {'content-type': 'application/json'}

response = requests.post(url, data=json.dumps(payload), 
                         headers=headers, verify=False)
Token = response.headers['x-auth-token ']

# read data
url2 = 'https://redfish-004-ipmi/redfish/v1/Chassis/1/'
Response = requests.get(url2, headers=headers, verify=False).json()

print (json.dumps(Response, indent=2))


                

Chassis data result




  "@odata.type": "#Chassis.1.0.0.Chassis",
  "SKU": "",
  "Name": "Computer System Chassis",
  "PartNumber": "",
  "AssetTag": "KS@KS@",
  "Links": {
    "ManagedBy": [
      {
        "@odata.id": "/redfish/v1/Managers/1"
      }
    ],
    "ContainedBy": {
      "@odata.id": "/redfish/v1/Chassis/Rack1"
    },
    "ComputerSystems": [
      {
        "@odata.id": "/redfish/v1/Systems/1"
      }
    ]
  },
  "SerialNumber": "S16068624511405",
  "@odata.id": "/redfish/v1/Chassis/1",
  "@odata.context": "/redfish/v1/$metadata#Chassis/Members/$entity",
  "Status": {
    "State": "Enabled",
    "Health": "OK"
  },
  "Thermal": {
    "@odata.id": "/redfish/v1/Chassis/1/Thermal"
  },
  "Power": {
    "@odata.id": "/redfish/v1/Chassis/1/Power"
  },
  "@Redfish.Copyright": "Copyright \u00a9 2014-2015 Distributed Management Task Force, Inc. (DMTF). All rights reserved.",
  "IndicatorLED": "Lit",
  "Oem": {
    "OemFan": {
      "@odata.type": "#OemFan.Chassis",
      "FanMode": "PUE2",
      "FanMode@Redfish.AllowableValues": [
        "Standard",
        "FullSpeed",
        "PUE2",
        "HeavyIO"
      ]
    }
  },
  "ChassisType": "RackMount",
  "Model": "",
  "Id": "1",
  "Manufacturer": "Supermicro"
}
{
  "@odata.type": "#Chassis.1.0.0.Chassis",
  "SKU": "",
  "Name": "Computer System Chassis",
  "PartNumber": "",
  "AssetTag": "KS@KS@",
  "Links": {
    "ManagedBy": [
      {
        "@odata.id": "/redfish/v1/Managers/1"
      }
    ],
    "ContainedBy": {
      "@odata.id": "/redfish/v1/Chassis/Rack1"
    },
    "ComputerSystems": [
      {
        "@odata.id": "/redfish/v1/Systems/1"
      }
    ]
  },
  "SerialNumber": "S16068624511405",
  "@odata.id": "/redfish/v1/Chassis/1",
  "@odata.context": "/redfish/v1/$metadata#Chassis/Members/$entity",
  "Status": {
    "State": "Enabled",
    "Health": "OK"
  },
  "Thermal": {
    "@odata.id": "/redfish/v1/Chassis/1/Thermal"
  },
  "Power": {
    "@odata.id": "/redfish/v1/Chassis/1/Power"
  },
  "@Redfish.Copyright": "Copyright \u00a9 2014-2015 Distributed Management Task Force, Inc. (DMTF). All rights reserved.",
  "IndicatorLED": "Lit",
  "Oem": {
    "OemFan": {
      "@odata.type": "#OemFan.Chassis",
      "FanMode": "PUE2",
      "FanMode@Redfish.AllowableValues": [
        "Standard",
        "FullSpeed",
        "PUE2",
        "HeavyIO"
      ]
    }
  },
  "ChassisType": "RackMount",
  "Model": "",
  "Id": "1",
  "Manufacturer": "Supermicro"
				

Change fan speed...



# Fan speed modes are:
# "Standard", #"FullSpeed"
# "PUE2", #"HeavyIO"

url2 = 'https://redfish-004-ipmi/redfish/v1/Chassis/1/'
headers = {'X-Auth-Token': Token}
payload2 = {"Oem": {
                "OemFan": {
                        "FanMode": "HeavyIO"
                          }
                   }
           }

output = requests.patch(url2, data=json.dumps(payload2), 
                        headers=headers, verify=False)
print output.status_code

                

Similar steps for various actions


"AssetTag": "DL5795959"

"IndicatorLED": "Lit"

+ some extras compared to IPMI

discover system topology (server, chassis, rack)

hard drive status / fault reporting

memory information

CPU information

Facebook's OpenBMC

Customized Linux distribution running on the BMC

includes a boot loader, a kernel, applications
  • open source, code is on github (2015)
  • used for servers, storage and network
  • no RMCP+
  • improved SSH support
  • REST API
  • one BMC supports multiple nodes
  • centralized authentication based on certificates

Wrap-up



Use your BMCs

Isolate your BMCs

Secure your BMCs

Update the firmware on your BMCs; if(issues): ping me

Have a go with the Redfish API

Dōmo arigatō gozaimasu

Questions?