Cosmic rays – the very name sounds like a futuristic beam weapon from a new sci-fi movie. But cosmic rays are quite real: powerful atomic particles hurled at near the speed of light by titanic explosions across the depths of intergalactic space. A hail of them, blown about by the solar wind and Earth’s magnetic shield, is constantly bombarding our homeworld from all directions. Primary cosmic rays are so energetic that they create showers of secondary particles when they penetrate the Earth’s atmosphere and strike atoms there.
Cosmic rays are more than scientific curiosities with a cool name. They pose health dangers to astronauts and spacecraft but can also have significant impacts on earthly things – including, it turns out, computer chips. And as such, the tiny energetic bits of matter present a whole new category of risk. They have caused fatal accidents, close calls for commercial airliners, and even changed election results.
Yes, it’s not just human hackers trying to fix elections that we need to be worried about, but also radiation from outer space. In a 2003 election in Belgium, one candidate wound up with more votes that seemed logically possible, yet there was no sign of hacking. During the recount, it was discovered that the candidate’s count was off by exactly 4,096 votes, which some bright person realized was was 2 to the 12th power. Which, it was reasoned, could only be the result of a miscount in the binary software due to a cosmic ray striking one specific bit and flipping it.
That’s all it takes — one highly-charged particle striking just the right spot on a tiny computer chip at just the right instant and unfortunate results can occur. In September 2007, a 2005 Camry accelerated out of control on an Oklahoma freeway, The car crashed on the on-ramp, killing one woman and seriously injuring another. The computer code governing the electronic throttle was discovered to be defective. Bugs in the code could be given a nudge by a single cosmic ray bit flip which the computer could not detect. As a result, the driver could and did lose control of the vehicle. And for that, the judge in this case awarded the injured parties $1.5 million.
The number of cosmic ray impacts increases dramatically with altitude. The higher a plane flies (up to around 60,000 feet) the more cosmic rays there are, which increases the chance of a significant event. Cosmic radiation has been ruled the cause in several cases of navigation problems in high-flying commercial jets. At normal cruising altitudes, LANL calculated that 2,000 such particles can penetrate each square yard of the aircraft’s surface every second, generally passing undetectably through the hull, electronics, engines, and the passengers. In 2008, a Quantas Airbus flying between Perth and Singapore had a failure where a computer sent incorrect data to the flight controls. The plane pitched severely downwards, injuring a third of the passengers and two-thirds of the crew. That incident was also blamed on a cosmic bit flip.
Data flipping has been known for a long time, often referred to as a “soft error” or a Single Event Upset (SEU). They were first detected as anomalies in electronic equipment during nuclear weapons testing, and soon after that in early spacecraft. As a result, chips for military and aerospace were hardened against radiation using insulation, shielding, and more rugged materials. Some can be many orders of magnitude more resistant than chips used in civilian terrestrial applications, with built error-detection capacity to find and fix errors.
Another way to correct is by having a set of three microprocessors all performing the same calculations. Disagreements between the devices would indicate an SEU. However, it should be noted that potential incidents of bit flipping by cosmic rays increase not only with the altitude the device, but also by decreasing the size of the chip. As integrated circuits get ever smaller according to Moore’s Law, they can be effected by less energetic particles. In other words, not just by primary cosmic rays, but the constant shower of secondary cosmic rays that they produce.
It’s conceivable that the problem could get even worse in the future, not only because of steadily shrinking chips but also due to increased radiation from space. The Earth’s weakening field from its on-going geomagnetic reversal has already produced an area – the South Atlantic Anomaly – where satellites and high-flying jets are adversely affected by more radiation getting through.
It is no accident that the examples cited above were all carefully-researched cases in highly-specific conditions. There may be many more errors happening all the time that go unnoticed. After all, it is an ancient joke among system administrators to attribute weird glitches and crashes in systems to “sunspots”. They might have been more correct than they could have imagined.
Maybe it’s not our brains that need to be wrapped in aluminum foil for protection from the rays, but our devices.
