I maintain a small project I care about, SearchHub, for some time now. It is a local search engine that indexes my bookmarks and lets me query external engines (crates.io, Wikipedia, etc).

I build an OCI image at each release for easy deployment, and until now I built locally from a Containerfile and tagged the image in my local registry.

I have a server that runs a bunch of services (including this blog), and I thought: why not self-host my own container registry there? That way I can push my images to it and pull from any machine without rebuilding.

The setup🔗

I run Debian on the server, with Docker installed for a few services. The official registry:2 image starts in one line:

sudo docker run -d --name registry -p 127.0.0.1:5050:5000 registry:2

Port 5050 is bound to localhost only, so I need a reverse proxy to reach it from outside. A small nginx block later:

server {
    server_name oci.vit.am;

    client_max_body_size 0;

    location / {
        proxy_pass http://127.0.0.1:5050;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 900;
    }
}

And Certbot for TLS. So far, so good.

Authentication🔗

I do not want just anyone to push to my registry. But I want everyone to be able to pull images (it is public). I protect writes with auth_basic:

location / {
    limit_except GET HEAD {
        auth_basic "Registry Login";
        auth_basic_user_file /etc/registry/auth/htpasswd;
    }

    proxy_pass http://127.0.0.1:5050;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 900;
}

This config says: “GET and HEAD are free, everything else requires a password.” Handy: pulls (GET) are anonymous, pushes (POST, PUT) are authenticated.

I create a user with htpasswd, reload nginx, test:

$ curl -s -o /dev/null -w "%{http_code}" https://oci.vit.am/v2/
200

$ curl -s -o /dev/null -w "%{http_code}" -X POST https://oci.vit.am/v2/search-hub/blobs/uploads/
401

Perfect: reads pass, writes need auth. I log in with buildah:

$ buildah login oci.vit.am
Username: paul
Password:
Login Succeeded!

I push:

$ buildah push oci.vit.am/search-hub:latest
Error: writing blob: initiating layer upload to /v2/search-hub/blobs/uploads/
        in oci.vit.am: authentication required

Wait, what?

The problem🔗

Yet I am logged in:

$ buildah login --get-login oci.vit.am
paul

I check with curl that my credentials work:

$ curl -s -o /dev/null -w "%{http_code}" \
  -u "paul:my-password" \
  -X POST https://oci.vit.am/v2/search-hub/blobs/uploads/
202

The credentials are fine. Buildah is just not sending them.

A quick --log-level debug tells me more:

DEBU GET https://oci.vit.am/v2/ → status 200
DEBU POST https://oci.vit.am/v2/search-hub/blobs/uploads/ → status 401
     WWW-Authenticate: Basic realm="Registry Login"

Buildah starts with GET /v2/, which is normal: it is the protocol ping, used to discover how to authenticate.

The issue is that nginx replies 200 without a WWW-Authenticate header. Buildah concludes the registry has no authentication and does not bother sending credentials for subsequent requests. When the POST hits nginx’s auth_basic and gets a 401, buildah does not retry with credentials: it already decided auth is unavailable and gives up.

The fix🔗

There are two options:

1. Protect reads too. Remove limit_except and put auth_basic on the entire location block. Downside: everyone must log in to pull.

2. Add the WWW-Authenticate header even on GET responses. That way buildah sees auth is available and sends credentials for requests that need them.

I went with the second one. The /v2/ block gets an add_header announcing Basic auth:

location /v2/ {
    add_header WWW-Authenticate 'Basic realm="Registry Login"' always;

    limit_except GET HEAD {
        auth_basic "Registry Login";
        auth_basic_user_file /etc/registry/auth/htpasswd;
    }

    proxy_pass http://127.0.0.1:5050;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 900;
}

location / {
    limit_except GET HEAD {
        auth_basic "Registry Login";
        auth_basic_user_file /etc/registry/auth/htpasswd;
    }

    proxy_pass http://127.0.0.1:5050;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_read_timeout 900;
}

Now the /v2/ ping replies:

200 + WWW-Authenticate: Basic realm="Registry Login"

Buildah sees the header, knows it must authenticate, and sends its credentials for subsequent requests. Anonymous pulls (GET, HEAD) are not blocked by nginx, they still go through without auth. Writes (POST, PUT) remain protected by auth_basic.

And this time:

$ buildah push oci.vit.am/search-hub:latest
Getting image source signatures
Copying blob sha256:523040b3fc983556b6486323e681ebfb7d35fe6685a3b1b5b4c9745417b5ed6f
Copying blob sha256:f4f8b983b714f130e2cff99176baa26352db6a55d3622e10ada40f2b4720a4eb

It works.

Why does buildah behave this way?🔗

This is standard Docker Registry V2 protocol behaviour. The client (buildah, podman, docker) starts with a ping to /v2/:

• If the server replies 401 with WWW-Authenticate, the client authenticates and retries.
• If the server replies 200 with WWW-Authenticate, the client knows it can authenticate for requests that need it.
• If the server replies 200 without WWW-Authenticate, the client considers the registry open and never sends credentials.

Our nginx config with limit_except made the ping reply 200 without WWW-Authenticate, putting buildah in the third case. Once it decided there is no auth, it never sends credentials again, even when the next request fails with 401.

The lesson: if you put a conditional auth_basic in front of a registry, make sure the /v2/ ping receives the WWW-Authenticate header. Otherwise, protocol-compliant clients will never send their credentials.