Percent-Encoding, explained: making sense of %20, %3F, and other URL hieroglyphs

In one sentence

URL encoding, officially called percent-encoding, is the process of translating characters that have special meaning or are invalid within a URL into a safe, universally understood format so they can be transmitted without causing confusion.

The problem it solves

In the primordial soup of the early web, life was simple. URLs—or more broadly, URIs (Uniform Resource Identifiers)—were designed to be a clean, predictable way to locate a resource. The architects, including Sir Tim Berners-Lee, built this system on a foundation of a limited character set: ASCII.

This worked great, as long as you only needed to point to http://example.com/reports/April.html. But what happens when things get more complicated?

Consider a URL's anatomy. It has parts: a scheme (http:), a host (example.com), a path (/search), and maybe a query string (?q=dogs&cats). Certain characters are structural directors in this play. The colon (:) separates the scheme. The slash (/) separates path segments. The question mark (?) kicks off the query parameters. The ampersand (&) separates one parameter from the next.

This is where the trouble starts. What if you want to search for the literal string "C++ & C#"? If you just slap that into a URL, you get .../search?q=C++ & C#. A web server sees this and gets hopelessly confused. It thinks the query is for "C++ ", and then it sees an ampersand and expects another key-value pair, but just gets a lonely " C#". Chaos. The intended meaning is lost.

Furthermore, some characters just aren't allowed. A space is a classic troublemaker. When is a space part of a filename, and when is it just a typo that a browser should ignore? And what about characters outside the basic English alphabet? The web is global! How do you put Résumé.pdf or 你好.html in a URL designed for ASCII?

Percent-encoding solves this entire class of problems. It provides an escape hatch, a way to say, "Hey, Mr. Web Server, the next character(s) aren't structural. Don't interpret them. They are literal data." It's the universal translator that ensures a URL means the same thing in a browser in Brazil as it does on a server in Berlin.

How it works under the hood

The "magic" behind percent-encoding is surprisingly straightforward. It's less of a magic trick and more of a simple substitution cipher that everyone has agreed to use.

The Cast of Characters: Reserved vs. Unreserved

First, you need to know which characters are cool and which are problematic. They fall into a few groups.

Character Type	Characters	When to Encode
Unreserved	`A-Z a-z 0-9 - _ . ~`	Never. These are the VIPs of the URL world. They are always safe.
Reserved	`: / ? # [ ] @ ! $ & ' ( ) * + , ; =`	Sometimes. These have special structural meaning. If you want to use them for their meaning (like `/` in a path), you don't encode. If you want to use them as literal data (like an `&` in a search query), you must encode.
Other (Unsafe)	(space), `< > " % { } \	^` and all non-ASCII characters

The key takeaway is context. The character ? is fine if it's the one ? that separates the path from the query string. But if you need a literal question mark inside a query parameter's value, you must encode it.

The Magic Trick: Percent + Hex

The encoding process is a simple three-step dance:

Pick a character you need to encode. Let's use the ampersand &.
Find its byte value using a standard character set. For the web, this standard is UTF-8. In UTF-8 (and its predecessor ASCII), the & character is represented by the decimal number 38.
Convert that number to two-digit hexadecimal and prepend a percent sign (%). Decimal 38 is 26 in hexadecimal.

So, & becomes %26.

Let's try a few more:

A space is decimal 32, which is hex 20. Encoded: %20.
A question mark (?) is decimal 63, which is hex 3F. Encoded: %3F.
The percent sign (%) itself is decimal 37, hex 25. So to encode a literal %, you write %25.

This system is brilliant because the percent sign itself is not an unreserved character, so a parser knows that whenever it sees a %, it should expect two hex digits to follow.

What about non-English characters?

This is where UTF-8 becomes critical. A simple ASCII character like A is one byte. But a character like the French é or the Chinese 好 is represented by multiple bytes in UTF-8. The encoding process is the same, just repeated for each byte.

Let's take é:

In UTF-8, é is represented by two bytes: C3 and A9 (in hex).
Encode each byte separately:
- C3 becomes %C3.
- A9 becomes %A9.
Combine them: é becomes %C3%A9.

The decoding process is the exact reverse. A browser or server sees %C3%A9, grabs the two bytes C3 and A9, runs them through a UTF-8 decoder, and gets back the beautiful é character.

Real-world stories

Theory is great, but let's see where the rubber meets the road.

The Case of the Disappearing Search Query

A junior dev, Maya, was building a search feature for a technical documentation site. Users could search for things like "C++", "promises & async/await", and so on. She built the search URL by simply concatenating strings: site.com/search?q= + userInput.

Things went haywire. A search for promises & async/await generated the URL .../search?q=promises & async/await. The server, however, only reported the search term as "promises ". The & was interpreted as a separator for a new parameter, async/await, which was discarded as it didn't have a key. Her search results were completely wrong.

The Lesson: Maya learned a cardinal rule of web development: always percent-encode any dynamic data being placed into a URL component. After she started encoding the user input, the URL correctly became .../search?q=promises%20%26%20async%2Fawait. The server now received the full, correct string, and the search worked perfectly.

The International Incident

An online store decided to feature a new product from a German partner: the "Fußball." The marketing team created a friendly URL for it: store.com/products/Fußball. On their modern browsers in the office, everything looked fine.

But launch day was a mess. Customer support tickets flooded in. Some users were getting "404 Not Found" errors. Others saw a URL that looked like .../products/Fu%C3%9Fball in their browser bar, while some saw .../products/FuÃball. The system was a patchwork of old and new components, and they weren't handling the non-ASCII character ß (Eszett) consistently. Some parts didn't encode it, some encoded it assuming UTF-8, and some legacy systems decoded it assuming a different character set, resulting in mojibake.

The Lesson: Relying on browsers and servers to "just handle" non-ASCII characters in URLs is a recipe for inconsistency. Proactively and consistently percent-encoding all non-unreserved characters using the UTF-8 standard ensures that your URLs are robust and work predictably across the entire web ecosystem, old and new.

The Double-Encoding Debacle

A team was building a single sign-on (SSO) system. The flow worked like this: service-a.com would redirect the user to sso.com/login, passing its own URL as a parameter so the user could be sent back after logging in. The redirect URL looked like this: sso.com/login?redirect_uri=https://service-a.com/dashboard?param=1.

The developer on service-a.com was smart and encoded the redirect_uri value, producing: sso.com/login?redirect_uri=https%3A%2F%2Fservice-a.com%2Fdashboard%3Fparam%3D1.

However, the web framework they used had a middleware layer that, "for security," automatically URL-encoded all outgoing query parameters. It saw the already-encoded string and encoded it again. The % in %3A was turned into %25, so %3A became %253A. The final URL was a garbled mess of double-encoding. When the user landed on sso.com, it decoded the URL once and got the single-encoded string, which it couldn't use as a redirect, breaking the login flow completely.

The Lesson: Be aware of your entire toolchain. Encode data at the point of creation, and ensure no other system down the line re-encodes it. Double-encoding is a common, head-scratching bug that turns a valid URL into useless junk.

Common mistakes and traps

Encoding the entire URL. Never do this. If you percent-encode https://example.com, you'll get something like https%3A%2F%2Fexample.com. This is no longer a valid URL; the scheme and authority parts are now just a meaningless jumble of characters. You must only encode the individual components that need it (like query parameter values or specific path segments).
Not encoding at all. The most frequent sin. Jamming raw user input or data with special characters directly into a URL string is asking for security holes (like Cross-Site Scripting) and broken functionality.
Forgetting about context. The & character is fine in a URL's path, but it's a reserved separator in the query string. Similarly for /. You don't need to encode reserved characters when they are being used for their special purpose.
Confusing + with %20. In the application/x-www-form-urlencoded content type (used by HTML forms), spaces are often encoded as a + sign in the query string. While many servers understand this, the official percent-encoding for a space is %20. Using %20 is unambiguous and works correctly in all parts of a URL, not just the query string. When in doubt, stick with %20.
Using an outdated character set. The web runs on UTF-8. If you encode your data using a different character set (like ISO-8859-1), then a server expecting UTF-8 will misinterpret the bytes and mangle your data. Always specify and use UTF-8.

Why it belongs on your radar

If you write code that touches a URL, you need to understand percent-encoding. It's not optional. You should think about it whenever you are:

Constructing a URL from variables or user input.
Making an API request with parameters in the URL.
Handling international characters in filenames, user profiles, or content that might appear in a URL.
Parsing a URL on the server-side to extract data.
Writing redirects or passing URLs as parameters to other services.

In short, percent-encoding is a fundamental piece of web plumbing. Ignoring it leads to buggy, insecure, and unreliable software. Knowing how it works is a mark of a professional web developer.

Go deeper

RFC 3986: The canonical specification for Uniform Resource Identifier (URI). Section 2 defines the character set and percent-encoding rules. It's the ultimate source of truth.
MDN Web Docs: encodeURIComponent(): A practical guide for JavaScript developers, explaining which function to use and why. The "See also" section links to other related encoding functions.
Wikipedia: Percent-encoding: A comprehensive and very readable overview of the concept, its history, and its various nuances.
W3C: Character encodings: A high-level introduction to why character encodings matter on the web, with UTF-8 as the hero of the story.