Heisenbugs; the Dark Side of Debugging

Some of the most difficult software bugs to track down are those that disappear when you try to study them… Heisenbugs! For example, when an issue is reproducible in a Release build, but disappears when you run the software in the Debugger.

There are many possible explanations for this which include differences in memory initialisation, timing, code optimisation, code and memory layout between Debug and Release builds. I won’t elaborate on the details here, as they are already very well covered on stack overflow.

The reason for this post is that we recently encountered an issue in some code which has the following behaviour:

  1. Has a reproducible bug in release mode (not an outright crash or exception, but fails to behave as expected).
  2. Runs perfectly well when started in the debugger, with no issues.
  3. When started without debugging, and the debugger is attached later, several first chance exceptions are seen.

My theory was that the exceptions weren’t being triggered when started from the Debugger because the memory initialisation (or other Debug/Release differences) were preventing the exceptions from occurring. However, there was some debate on this point, with someone else suggesting that attaching the Debugger later was causing spurious exceptions. Their suggestion was that if these exceptions were genuine, they would also occur when started in the Debugger.

I’ve seen this debate crop up frequently enough, that I thought it would be useful to create a very simple example to demonstrate an exception that reproduces in Release, but works fine in Debug. Also, to confirm whether the same exception would reproduce when the Debugger is attached to a running process.

Here is a first attempt at creating a very small Heisenbug demonstrator in C++, which has a deliberate access violation:

#include <iostream>

void main() {
    int *x = new int[3];
    if (x[1] != x[2]) delete [] x;
    std::cin.get();
    delete [] x;
}

When a Release build is run in Visual Studio 2013, it demonstrates the following behaviour:

  1. When started without the Debugger and a key is pressed, it causes an exception.
  2. When started under the Debugger and a key is pressed, it exits cleanly and works fine.
  3. When started without the Debugger, and the Debugger is later attached and a key is pressed, it causes an exception.

If you run this a few times, you might find that Windows enables the “Fault Tolerant Heap” for the executable, and then you won’t be able to reproduce the exceptions. The solution is to disable the Fault Tolerant Heap for your executable, by going to these keys:

HKLM\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers\heisenbug.exe

HKCU\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Layers\heisenbug.exe

Then delete the Fault­Tolerant­Heap entry on both of these keys. In my case, only the HKLM version was needed.

Returning to the code to explain how this works, here’s a commented version, explaining each line:

// create an array of 3 integers, without initialisation
int *x = new int[3];

// compare elements 1 and 2 (which are not initialised)
// if they are different, delete the array and leave
// the invalid pointer in x
if (x[1] != x[2]) delete [] x;

// wait for a key press
std::cin.get();

// delete the array (potential access violation)
delete [] x;

There are two intentional bugs in this particular line of code:

if (x[1] != x[2]) delete [] x;

The first bug is that we are accessing uninitialised memory, because the array x has not beeen initialised. When we compare x[1] and x[2] the result is unpredictable. In a Release build there’s a good chance (but not absolute certainty) that these values will be different, and it will delete the array. Second, after deleting the array, the variable x still contains the (now invalid) pointer address. Later on, when this final line of code executes, it will access an invalid pointer, usually causing an access violation exception:

delete [] x;

However, when running in the Debugger, it’s very likely that the memory will have been initialised by the Debugger, and x[1] and x[2] will usually contain the same value. This prevents the pointer from being deleted, and avoids the access violation. The code runs without issue when being debugged.

This serves as a very simple example of a bug which only appears in Release, and disappears in the Debugger, due to accessing uninitialised memory (as mentioned above, this is only one of many possible causes of such an issue).

Abarth 500 Fuse Box / K1S Dashcam Wiring

I recently fitted a K1S Dashcam (Koonlung K1S) to an Abarth 500, and wanted to have a tidy install with no wires showing if possible. To avoid having it plugged into the 12V lighter socket, I needed to find a suitable place to wire it into the dashboard fuse box. Although the fuse box is described in the manual, it doesn’t indicate which fuses are permanently live, and which are switched live. This information is not readily available on the interwebs either, so I figured it out with a meter and made some partial notes in the table below.

FuseAmpsDutyDescription
F127.5Right dipped headlamp power supply
F137.5Left dipped headlight and headlight alignment control unit power supply
F327.5ConstantFront, rear and luggage compartment roof lights
F3610Diagnosis socket, radio, climate control, EOBD
F375Brake light switch, instrument panel node
F3820Door central locking
F4315Windscreen/rear window washer pump
F4720ConstantDriver's side electric windows
F4820ConstantPassenger side electric window
F495SwitchedParking sensor, control backlighting, electric mirrors
F507.5Airbag node
F517.5SwitchedRadio switch, Blue&Me, climate control, brake lights, clutch
F535Instrument panel node

Obviously, there are a lot of fuses there which are “safety critical” such as the Airbag, Brake Lights etc. which are best avoided. Also, the fuse layout is probably the same on the standard Fiat 500, but you know… they might not be, so use this information at your own risk!

The picture of the fuse box in the manual is upside down (for right hand drive models) so here’s a rotated version with the fuse ratings added in blue. The Switched (S) and Constant (C) supply fuses are (partially) labelled in red:

Abarth Fuse Box

To avoid cutting any wiring, I used one of these mini “add-a-circuit” widgets, which allow you make another connection to an existing fuse:

2016-03-25_add_a_circuit_empty

They need to be used with some care, because you could fit the original and new fuses in the wrong places, and fit it in the wrong orientation. As there were no instructions with it, I used the resistance range on my meter to double-check the wiring before fitting the fuses.

In the picture above/below, the left blade is the input side, which needs to go to the live supply in the fuse box. The right hand pin nearest the wire is the output, after the fuse. The original fuse goes in the lower two holes. The new fuse for your added circuit goes in the top two holes, and the output of that fuse is the red wire. Pretty self explanatory really:

2016-03-25_add_a_circuit_fuses

When installed in one orientation (12V input to left blade as described above), the input goes through F1 to item A and the input goes through F2 to item B. If either fuse blows, the other item should still have power. However, when installed in the reverse orientation, note that the input will go through F1 to item A but will go through both F1 and F2 in series to item B. So you need to consider the orientation when installing it.

The wire projecting from the side of the “add-a-circuit”, and the fuses projecting sideways can make it impossible to fit in some positions and orientations (e.g. if they interfere with the side of the fuse box). This can limit your choices on which fuses it can be connected to.

Here’s a shot of it fitted in situ on the fuse box. In this case, it’s inserted into F49 (Parking Sensor, control backlighting, electric mirrors). Using a meter, I found that the left terminal in F49 was live with the fuse removed, hence the output wire faces right when installed:

2016-03-25_abarth_fusebox_k1s

Caution: whereas the left terminal of F49 was live, I found that it was the right terminal of F47 and F48 that was live, so it isn’t the same for every fuse.

For the ground connection, I used an eyelet crimp terminal under one of the two bolts which secures the bonnet release handle to the chassis. This is a very short distance from the fuse box, so easy to run a wire.