An 11-Byte Message Can Quietly Crash Servers Running Old OpenSSL

Okta's Red Team has disclosed HollowByte, a denial-of-service flaw in OpenSSL that lets a remote attacker exhaust a server's memory using an 11-byte message and no authentication at all — before any encryption handshake even completes.

The bug lives in how older OpenSSL versions handle the very first message of a TLS connection. That message declares its own size in a 3-byte header, and vulnerable OpenSSL versions trust that number immediately, allocating up to 131 KB of memory before a single byte of actual data arrives. An attacker who never sends the promised data leaves the server waiting indefinitely.

What makes HollowByte worse than a typical connection-flooding attack is what happens after the connection drops. On Linux systems using the standard glibc memory manager, freed memory isn't always returned to the operating system — it's kept in reserve for reuse. By varying the claimed message size on each attempt, an attacker fragments that reserved memory until the server can no longer reuse it efficiently, so total memory use keeps climbing even after the attack traffic stops. Okta's testing showed a server with 1 GB of RAM crashed entirely, killed by the operating system after 547 MB became permanently locked up in unusable fragments; a 16 GB server had a quarter of its memory tied down the same way, all while staying under the connection limits that normally catch this kind of abuse.

Because OpenSSL underpins web servers like Apache and NGINX, language runtimes including Node.js, Python and PHP, and databases such as MySQL and PostgreSQL, the exposure is broad. OpenSSL fixed the issue on June 9 in versions 4.0.1, 3.6.3, 3.5.7, 3.4.6 and 3.0.21 — but classified it as a routine hardening change rather than a security vulnerability, so it received no CVE identifier and no advisory. That decision means many vulnerability scanners and patch-tracking systems built around CVE numbers won't flag servers as at risk, even though they are.

Administrators should confirm they're running a patched OpenSSL version and restart any service that loaded the library before updating, since a running process keeps using the old code in memory until it's restarted.