nginx / no realip in logs / bug

Recently we were doing a vulnerability scan on an external endpoint of a website. During this time, we noticed simultaneous suspicious activity coming from inside our internal network. What's the deal?

To make a long story short: we use a proxy (like HAProxy) in front of our webservers, and it speaks the Proxy Protocol to pass along the original client's IP address. That way, when a request hits our internal nginx server, the logs show the real client's IP, not the IP of the reverse proxy that actually made the connection.

But during our scan, some failed requests showed up in the logs with the proxy's IP address — not the external client's. This made us pause: “Wait… is someone inside our network scanning this site too?”

We figured it out quickly, but it was still unsettling. And in our view, this is a bug.

Reproducing

Step one: fetch the latest nginx.

$ git clone https://github.com/nginx/nginx
...

Configure and build it:

$ ./auto/configure --with-debug --with-http_realip_module
...

$ make -j$(nproc)
...

$ ./objs/nginx -V
nginx version: nginx/1.29.0
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
configure arguments: --with-http_realip_module

Set up a basic config in ./conf/nginx_realip_bug.conf:

worker_processes  1;
error_log stderr info;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  text/plain;

    set_real_ip_from  127.0.0.0/8;
    real_ip_header    proxy_protocol;

    access_log /dev/stderr;

    server {
        listen       8080 proxy_protocol_always;
        server_name  localhost;
        location / {
            return 200 "Hello world\n";
        }
    }
}

Run nginx in the foreground:

$ ./objs/nginx -p $PWD -c conf/nginx_realip_bug.conf -g 'daemon off;'
2025/05/29 22:59:23 [notice] 154902#0: using the "epoll" event method
2025/05/29 22:59:23 [notice] 154902#0: nginx/1.29.0
2025/05/29 22:59:23 [notice] 154902#0: built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
...

Do a simple request:

$ printf '%s\r\n' \
    'PROXY TCP4 1.2.3.4 127.127.127.127 80 8080' \
    'GET /foo HTTP/1.0' '' |
    nc 127.0.0.1 8080 -w2

HTTP/1.1 200 OK
Server: nginx/1.29.0
Date: Fri, 30 May 2025 19:36:40 GMT
Content-Type: text/plain
Content-Length: 12
Connection: close

Hello world

This could use some explanation. The GET /foo HTTP/1.0 is the oldest/most basic HTTP request, followed by two sets of CR LF. The printf '%s\r\n' provides those.

We could have used curl --path-as-is --haproxy-protocol localhost:8080/foo, instead of the printf and nc combination. But curl doesn't let us set values (like 1.2.3.4), and for this example we need them to be visibly different.

The location / in the configuration accepts everything below / — like /foo — and returns a 200 OK status code and the Hello world message.

Proxy protocol

The Proxy Protocol — originally proposed by Willy Tarreau — is a simple way to preserve information about the real client when traffic passes through one or more proxies.

Shown here is the v1 version:

PROXY TCP4 1.2.3.4 127.127.127.127 80 8080

This line tells the receiving server that the original IPv4 client (1.2.3.4) is connecting to the destination IP (127.127.127.127) — from port 80 to port 8080. A reverse proxy injects this line at the start of the connection, so the upstream (or "origin") server can see who actually made the request, even though it's technically connected to the proxy.

Yes, HTTP has X-Forwarded-For, but that header is less reliable: it depends on careful trust boundaries, can be spoofed, and is harder to manage across non-HTTP protocols.

This is all explained in detail at proxy-protocol.txt.

We use reverse proxies to protect our internal network from the big bad internet. But for the machines behind those proxies, the Proxy Protocol is how they know who's really knocking.

Real IP in logs

With the provided nginx config, the first GET /foo from above shows up in the logs as:

1.2.3.4 - - [30/May/2025:21:36:40 +0200] "GET /foo HTTP/1.0" 200 12 "-" "-"

Let's do another request, with the funky, but somewhat legal, /foo/../foo path:

$ printf '%s\r\n' \
    'PROXY TCP4 1.2.3.4 127.127.127.127 80 8080' \
    'GET /foo/../foo HTTP/1.0' '' |
    nc 127.0.0.1 8080 -w2

HTTP/1.1 200 OK
Server: nginx/1.29.0
Date: Fri, 30 May 2025 19:36:57 GMT
Content-Type: text/plain
Content-Length: 12
Connection: close

Hello world

We still get a 200 status. And this log line:

1.2.3.4 - - [30/May/2025:21:36:57 +0200] "GET /foo/../foo HTTP/1.0" 200 12 "-" "-"

So far so good. Now make it worse by trying to escape from the root path with /foo/../../foo:

$ printf '%s\r\n' \
    'PROXY TCP4 1.2.3.4 127.127.127.127 80 8080' \
    'GET /foo/../../foo HTTP/1.0' '' |
    nc 127.0.0.1 8080 -w2

HTTP/1.1 400 Bad Request
Server: nginx/1.29.0
Date: Fri, 30 May 2025 19:37:11 GMT
Content-Type: text/html
Content-Length: 157
Connection: close

<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.29.0</center>
</body>
</html>

Boom! Rejected. That's okay. That's good in fact.

Proxy IP in logs

And the logs?

2025/05/30 21:37:11 [info] 175878#0: *3 client sent invalid request while reading client request line, client: 127.0.0.1, server: localhost, request: "GET /foo/../../foo HTTP/1.0"
127.0.0.1 - - [30/May/2025:21:37:11 +0200] "GET /foo/../../foo HTTP/1.0" 400 157 "-" "-"

Two log lines this time. But here's the problem: they show the proxy's IP (127.0.0.1) instead of the real client IP (1.2.3.4). Not nice.

There's a simple technical reason. nginx rejects the malformed request before any modules get a chance to run — including the realip module that rewrites IP addresses.

This isn't limited to bad HTTP requests either. TLS handshake errors and other early-stage failures can also bypass the modules, leading to the same issue: logs that show the proxy's IP instead of the real one.

But just because there's an explanation doesn't mean it's okay.

Bug filed at nginx/nginx#717.

Let's see what upstream has to say.