In NetHack, backstab refers to the extra damage done by Rogues on fleeing monsters; attacks doing backstab damage are sometimes called sneak attacks. It is calculated as 1dxplevel. Only melee attacks count toward backstab damage.
A basic means of utilizing backstabs is to inscribe Elbereth with a monster next to you, then step off to attack the monster as it turns to flee. This can be repeated ad infinitum, with the only limitation being your patience, but is probably not worth your time against monsters that you can kill easily. Musical instruments, mirrors, and all the other methods of scaring can be used to set up easy backstabs.
In NetHack 3.4.3 and variants based on it, any weapon's damage against a fleeing monster while not wielding two weapons would qualify for the backstab bonus, including thrown objects such as daggers. This was changed in 3.6.0, with the combined bonuses of multiple flying daggers being seen as too powerful.
In dNetHack, in addition to Rogues, backstab dice are available to Drow Anachrononauts, Convicts (only when wielding a spoon), any character with Andromalius bound, and lightsaber-wielders using the Juyo form. Backstab dice apply when the target is fleeing, trapped, unmoving, sleeping, stunned, confused, or blinded. With the Lifehunt Scythe, players can also backstab if the monster doesn't know their location.
In xNetHack, backstab damage is limited by weapon skill; it can't go above d2/d4/d10/d20/d30 for Restricted/Unskilled/Basic/Skilled/Expert. Backstab damage applies when the monster is fleeing, trapped, frozen, unmoving, sleeping, stunned, confused, or blinded.