nginx / no realip in logs / bug
Recently we were doing a vulnerability scan on an external endpoint of a website. During this time, we noticed simultaneous suspicious activity coming from inside our internal network. What's the deal?
To make a long story short: we use a proxy (like HAProxy) in front of our webservers, and it speaks the Proxy Protocol to pass along the original client's IP address. That way, when a request hits our internal nginx server, the logs show the real client's IP, not the IP of the reverse proxy that actually made the connection.
But during our scan, some failed requests showed up in the logs with the proxy's IP address — not the external client's. This made us pause: “Wait… is someone inside our network scanning this site too?”
We figured it out quickly, but it was still unsettling. And in our view, this is a bug.
Reproducing
Step one: fetch the latest nginx.
$ git clone https://github.com/nginx/nginx
...
Configure and build it:
$ ./auto/configure --with-debug --with-http_realip_module
...
$ make -j$(nproc)
...
$ ./objs/nginx -V
nginx version: nginx/1.29.0
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
configure arguments: --with-http_realip_module
Set up a basic config in ./conf/nginx_realip_bug.conf
:
worker_processes 1;
error_log stderr info;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type text/plain;
set_real_ip_from 127.0.0.0/8;
real_ip_header proxy_protocol;
access_log /dev/stderr;
server {
listen 8080 proxy_protocol_always;
server_name localhost;
location / {
return 200 "Hello world\n";
}
}
}
Run nginx in the foreground:
$ ./objs/nginx -p $PWD -c conf/nginx_realip_bug.conf -g 'daemon off;'
2025/05/29 22:59:23 [notice] 154902#0: using the "epoll" event method
2025/05/29 22:59:23 [notice] 154902#0: nginx/1.29.0
2025/05/29 22:59:23 [notice] 154902#0: built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
...
Do a simple request:
$ printf '%s\r\n' \
'PROXY TCP4 1.2.3.4 127.127.127.127 80 8080' \
'GET /foo HTTP/1.0' '' |
nc 127.0.0.1 8080 -w2
HTTP/1.1 200 OK
Server: nginx/1.29.0
Date: Fri, 30 May 2025 19:36:40 GMT
Content-Type: text/plain
Content-Length: 12
Connection: close
Hello world
This could use some explanation. The GET /foo HTTP/1.0
is the
oldest/most basic HTTP request, followed by two sets of CR LF. The
printf '%s\r\n'
provides those.
We could have used
curl --path-as-is --haproxy-protocol localhost:8080/foo
, instead of theprintf
andnc
combination. But curl doesn't let us set values (like 1.2.3.4), and for this example we need them to be visibly different.
The location /
in the configuration accepts everything below /
—
like /foo
— and returns a 200 OK
status code and the
Hello world
message.
Proxy protocol
The Proxy Protocol — originally proposed by Willy Tarreau — is a simple way to preserve information about the real client when traffic passes through one or more proxies.
Shown here is the v1 version:
PROXY TCP4 1.2.3.4 127.127.127.127 80 8080
This line tells the receiving server that the original IPv4 client (1.2.3.4) is connecting to the destination IP (127.127.127.127) — from port 80 to port 8080. A reverse proxy injects this line at the start of the connection, so the upstream (or "origin") server can see who actually made the request, even though it's technically connected to the proxy.
Yes, HTTP has X-Forwarded-For, but that header is less reliable: it depends on careful trust boundaries, can be spoofed, and is harder to manage across non-HTTP protocols.
This is all explained in detail at proxy-protocol.txt.
We use reverse proxies to protect our internal network from the big bad internet. But for the machines behind those proxies, the Proxy Protocol is how they know who's really knocking.
Real IP in logs
With the provided nginx config, the first GET /foo
from above
shows up in the logs as:
1.2.3.4 - - [30/May/2025:21:36:40 +0200] "GET /foo HTTP/1.0" 200 12 "-" "-"
Let's do another request, with the funky, but somewhat legal,
/foo/../foo
path:
$ printf '%s\r\n' \
'PROXY TCP4 1.2.3.4 127.127.127.127 80 8080' \
'GET /foo/../foo HTTP/1.0' '' |
nc 127.0.0.1 8080 -w2
HTTP/1.1 200 OK
Server: nginx/1.29.0
Date: Fri, 30 May 2025 19:36:57 GMT
Content-Type: text/plain
Content-Length: 12
Connection: close
Hello world
We still get a 200 status. And this log line:
1.2.3.4 - - [30/May/2025:21:36:57 +0200] "GET /foo/../foo HTTP/1.0" 200 12 "-" "-"
So far so good. Now make it worse by trying to escape from the root path
with /foo/../../foo
:
$ printf '%s\r\n' \
'PROXY TCP4 1.2.3.4 127.127.127.127 80 8080' \
'GET /foo/../../foo HTTP/1.0' '' |
nc 127.0.0.1 8080 -w2
HTTP/1.1 400 Bad Request
Server: nginx/1.29.0
Date: Fri, 30 May 2025 19:37:11 GMT
Content-Type: text/html
Content-Length: 157
Connection: close
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.29.0</center>
</body>
</html>
Boom! Rejected. That's okay. That's good in fact.
Proxy IP in logs
And the logs?
2025/05/30 21:37:11 [info] 175878#0: *3 client sent invalid request while reading client request line, client: 127.0.0.1, server: localhost, request: "GET /foo/../../foo HTTP/1.0"
127.0.0.1 - - [30/May/2025:21:37:11 +0200] "GET /foo/../../foo HTTP/1.0" 400 157 "-" "-"
Two log lines this time. But here's the problem: they show the proxy's IP (127.0.0.1) instead of the real client IP (1.2.3.4). Not nice.
There's a simple technical reason. nginx rejects the malformed request before any modules get a chance to run — including the realip module that rewrites IP addresses.
This isn't limited to bad HTTP requests either. TLS handshake errors and other early-stage failures can also bypass the modules, leading to the same issue: logs that show the proxy's IP instead of the real one.
But just because there's an explanation doesn't mean it's okay.
Bug filed at nginx/nginx#717.
Let's see what upstream has to say.