Integer overflow is what occurs when a numeric variable exceeds its maximum capacity. Generally what happens, due to two's complement arithmetic, is that the variable wraps to the opposite end of the range. So for a signed variable, a negative value just beyond the range would become a large positive value, and vice versa. An unsigned variable that exceeds its capacity would become a small positive value, or zero. Subtracting too much from a small unsigned variable would produce a large value.
A signed 8-bit variable (which is what AC uses) can store -128 to 127 inclusive. If your AC is -126 and you put on a piece of armor that would reduce your AC by five points, then your AC would become -131. -131 is too large for a signed 8-bit variable, so your new AC would be 124. 124 AC is of course extremely bad, considering a naked human has 10 AC. Thankfully, this doesn't happen in the case of AC.
Some integers in NetHack, like AC and alignment record, have protection against overflow. One way you can detect whether an overflow would occur is if subtracting from a negative value produces a positive value, or adding to a positive value produces a negative value. In the case of AC, the code calculates your current AC in a larger variable (varies, but most likely 32 bits), and then checks if the smaller AC variable (8 bits) would overflow.
Items with enchantments can go from -128 to +127 without overflowing.
It is also possible to cause integer overflow of the weight of containers or the player's inventory, causing it to become negative. 4294968 loadstones or 357914 boulders in a chest will cause an overflow to about -2147483647, or -(2^31). This tends to cause the encumbrance status to go away. If inventory weight display is enabled, the player will see that the inventory weight and they will be able to pick up any object, provided that they have an inventory letter free. The player can also wish for objects such as boulders and not have them drop to the floor. Overflowing the player's inventory weight is only possible by wishing for cursed loadstones, and this usually can only be done in wizard mode. It is best to do this when the player is already carrying 52 different items so the loadstone will take up the '#' slot.