What are Machine Check Exceptions (or MCE)?

A machine check exception is an error detected by your system’s processor. There are 2 major types of MCE errors, a notice or warning error, and a fatal exception. The warning will be logged by a “Machine Check Event logged” notice in your system logs, and can be later viewed via some Linux utilities. A fatal MCE will cause the machine to stop responding and the details of the MCE will be printed out to the system’s console.

What causes MCE errors?

There most common reason for MCE events to occur are:

• Memory errors or Error Correction Code (ECC) problems
• Inadequate cooling / processor over-heating
• System bus errors
• Cache errors in the processor or hardware

How do I find out what the errors mean?

If you see the message “Machine Check Events logged” on your console or in your system logs, then you can run the mcelog command to read the message from the kernel. Once you run mcelog you will not be able to re-run it to see the error, so it’s best to output the text to a file so you can further analyze it. For example:

root@localhost:/root> /usr/sbin/mcelog > mcelog.out

Some systems do this for you on a regular basis and send the output to the file /var/log/mcelog . So if you see the “Machine Check Events logged” message but mcelog does not return any data, please look /var/log/mcelog.

The output received may not always be easy to understand. If you have any questions about the decoded error message please create a support ticket and we will help analyze the problem.

What if I get a fatal machine check event that causes my machine to stop responding?

These errors are almost always caused by faulty hardware. Please capture the MCE message and you can later run it through the mcelog program once the machine is back up. Here’s an example of a message you might see:

CPU 1: Machine Check Exception:                4 Bank 4:  f600200137080813
TSC b0ce27165dd3 ADDR 180ee1b40

Paste or type the error message into a file, and then run it through the mcelog for example:

root@localhost:/root> /usr/sbin/mcelog --k8 --ascii < myerror

Use the –k8 option if you are using an AMD Opteron or Athlon 64 processor, or substitute it for –p4 for a Pentium 4 or Xeon. Here is the output from the previous MCE error:

HARDWARE ERROR. This is *NOT* a software problem!
Please contact your hardware vendor
CPU 1 4 northbridge TSC b0ce27165dd3
 Northbridge Chipkill ECC error
 Chipkill ECC syndrome = 3700
 bit32 = err cpu0
 bit45 = uncorrected ecc error
 bit57 = processor context corrupt
 bit61 = error uncorrected
 bit62 = error overflow (multiple errors)
 bus error 'local node origin, request didn't time out
 generic read mem transaction
 memory access, level generic'
STATUS f600200137080813 MCGSTATUS 4

This indicates that an uncorrected ECC error occurred. This indicates that one of your memory modules has failed. For further analysis please submit a support ticket with the complete MCE error message and the output of mcelog.