Variable-Length Subnet Masking (VLSM) amounts to "subnetting subnets," which means that VLSM allows network engineers to divide an IP address space into a hierarchy of subnets of different sizes, making it possible to create subnets with very different host counts without wasting large numbers of addresses.
A subnet mask defines the size of the subnet (the number of host addresses in the subnet). Fixed-Length Subnet Masking (FLSM) creates subnets all the same size. But where some subnets will have many hosts and some have few, FLSM results in some subnets having many orphaned addresses, or some sets of hosts being too big to fit into a subnet. Where VLSM is enabled, a large subnet can be divided into a set of smaller sub-subnets, which can be used to handle smaller sets of hosts.
For example, consider a traditional Class C address space like 192.168.1.0 and an organization with four groups of computers: the data center with 75 hosts; the call center with 50; the operations floor with 25; and the executive floor with 20. Under fixed subnetting, dividing the 255 host addresses available into four subnets would support only 62 hosts each, not meeting the needs of the data center and vastly oversupplying addresses for operations and the execs. Using VLSM, the space is first split in 2, with each subnet able to address 126 hosts. One subnet covers the data center. The other is split in two, supplying two sub-subnets of 62 hosts. One covers the contact center, the other is split in two once more, creating two 30-host sub-sub-subnets, to cover operations and executives.
In order to use VLSM, a network administrator must use a routing protocol that supports it, such as Routing Information Protocol v2 (RIPv2), Open Shortest Path First (OSPF), Intermediate System-to-Intermediate System (IS-IS), Enhanced Interior Gateway Routing Protocol (EIGRP) and Border Gateway Protocol (BGP).
VLSM is similar in concept and intent to Classless Inter Domain Routing (CIDR), which allows a single Internet domain to have an address space that does not fit into traditional address classes. VLSM was originally defined in IETF RFC 1812.
This video tutorial provides a simple use case for how (and why) to use VLSM.