Docker 94 🐳 Debugging Ephemeral / Crashed Containers: Debugging Distroless Containers with Ephemeral Containers

ByKandZ October 21, 2026October 21, 2026

⏱️ Estimated read: 14 min | |

Distroless images exist to reduce attack surface: they contain the application binary and its runtime dependencies, and nothing else. No shell, no package manager, no curl, no strace. When a distroless container crashes or misbehaves, the standard debugging reflex—docker exec -it <container> /bin/sh—fails because there is no shell to execute . The ephemeral container pattern exists to solve this: attach a separate container with debugging tools to the target’s namespaces, inspect the target from the outside, and remove the debug container without modifying the production image or restarting the workload.

Key point: An ephemeral container is not a modification of the target container. It is a second container that shares the target’s namespaces (network, PID, or both), running a different image that contains the tools the target lacks. The target’s filesystem remains its own; the debug container’s filesystem is separate. This is why the pattern works for distroless images and crashed containers alike.

Why distroless debugging requires a different approach

The shell absence problem. A distroless image built from gcr.io/distroless/static-debian12 contains no /bin/sh, no /bin/bash, and no package manager. docker exec or kubectl exec into the container fails with “executable file not found in $PATH” . The failure is not a configuration error; it is the intended behavior of the image.

The crashed container problem. If the container has already exited, there is no running process to attach to. docker exec requires a running container. The container’s filesystem still exists on disk, and its logs are still available, but the process that needs debugging is gone. The approach for a crashed container is different: inspect the exit code, the logs, and the filesystem, then run a new container with the same image and an overridden entrypoint to reproduce the failure .

The namespace sharing problem. A debug container that does not share the target’s namespaces sees its own network stack, its own process tree, and its own filesystem. Running ps inside it shows only the debug container’s processes. Running curl localhost:8080 tests the debug container’s localhost, not the target’s. The --target flag (in Kubernetes) or --pid=container: and --net=container: flags (in Docker) are what make the debug container useful .

The ephemeral nature problem. An ephemeral container is created through a special API handler, not through the pod spec. It cannot be added with kubectl edit, it does not restart when it exits, and it cannot be removed from the pod’s spec. It runs until the session ends, then stops . This is intentional: the debug container is a temporary inspection tool, not a permanent part of the workload.

a. Docker: The Sidecar Debug Pattern

Docker does not have a built-in docker debug command in the open-source CLI. Docker Desktop 4.27+ introduced docker debug, which attaches a toolbox container to the target’s namespaces, but this is a Desktop feature, not available on plain Docker Engine or Linux CI servers . The portable alternative is the sidecar pattern: launch a new container with a tools image and share the target’s PID and network namespaces.

# Get the target container's ID
TARGET_ID=$(docker ps -q --filter "name=my-distroless-app")

# Launch a debug container sharing the target's PID and network namespaces
docker run -it --rm \
  --pid=container:$TARGET_ID \
  --net=container:$TARGET_ID \
  --cap-add=SYS_ADMIN \
  nicolaka/netshoot

The --pid=container: flag makes the debug container see the target’s processes. Running ps aux inside the debug container shows the target’s application process as PID 1, along with any child processes . The --net=container: flag makes the debug container share the target’s network stack, so localhost:8080 inside the debug container is the target’s port 8080. The --cap-add=SYS_ADMIN is required for strace and other tracing tools that need to attach to processes in the shared PID namespace.

The nicolaka/netshoot image includes curl, dig, tcpdump, netstat, ss, strace, and other network and process debugging tools. A smaller alternative is busybox, which provides sh and basic utilities but lacks network-specific tools . The choice depends on what needs to be inspected.

# Inside the netshoot debug container
ps aux                    # Shows target's processes
ss -tlnp                  # Shows target's listening ports
curl localhost:8080       # Tests target's local service
strace -p 1               # Traces target's PID 1 (requires SYS_ADMIN)

The filesystem of the target is not directly mounted in the debug container. To inspect the target’s files, the /proc/<pid>/root path provides access to the target’s root filesystem from the debug container’s perspective, provided the debug container has the necessary permissions.

b. Kubernetes: kubectl debug with Ephemeral Containers

Kubernetes has first-class support for ephemeral containers through kubectl debug. The command attaches a new container to an existing pod, with the --target flag specifying which container’s namespaces to share .

# Attach a debug container to a distroless pod
kubectl debug -it my-distroless-pod \
  --image=busybox \
  --target=app

The --target=app flag makes the ephemeral container share the PID namespace of the app container. The output confirms the targeting: “Targeting container ‘app’. If you don’t see processes from this container it may be because the container runtime doesn’t support this feature” . Once inside, ps aux shows the target’s processes, and netstat -lntu shows the target’s listening ports.

Process namespace sharing is not always enabled by default. If the ps output inside the debug container does not show the target’s processes, the pod may need shareProcessNamespace: true in its spec. This is a pod-level setting that affects all containers in the pod, and changing it requires recreating the pod . The --target flag is supposed to handle this automatically by sharing the PID namespace of the specified container, but the behavior depends on the container runtime’s support for the feature.

For a container that has already crashed, kubectl debug offers a different mode: --copy-to. This creates a copy of the pod with a new container that has the same image but a different command, allowing the debugger to inspect the environment without affecting the original workload .

# Create a copy of the pod with a debug container
kubectl debug -it my-pod \
  --image=busybox \
  --copy-to=my-debug-pod \
  --share-processes

The copied pod is a separate resource. It does not inherit the original pod’s labels, so it is not selected by the original Service. It remains in the cluster until deleted, which means cleanup is a manual step .

c. Crashed Containers: Inspecting State Before Attaching

When a container has already exited, the debug container pattern is not the first step. The first step is inspecting what the container left behind: its exit code, its logs, and its configuration.

# Check the exit code
docker inspect --format='{{.State.ExitCode}}' my-crashed-container

# View the logs from the last run
docker logs my-crashed-container

# Inspect the container's configuration and mounts
docker inspect my-crashed-container

The exit code narrows the failure class. Exited (0) means the process completed successfully but the container was expected to keep running. Exited (1) means an application error. Exited (137) means the process was killed by SIGKILL, often because of an OOM condition. Exited (143) means the process received SIGTERM, which is the normal shutdown signal .

If the container is distroless and crashed, the logs may be empty because the application buffered its output and the buffer was lost when the process died. In that case, the next step is running a new container with the same image but an overridden entrypoint that keeps the container alive, then attaching a debug container to it.

# Run the same image with a shell-like entrypoint (if the image has one)
# For distroless, this will fail, so use a debug sidecar instead
docker run -d --name debug-target my-distroless-image

# Attach a debug container to the running target
docker run -it --rm \
  --pid=container:debug-target \
  --net=container:debug-target \
  nicolaka/netshoot

If the image has no shell at all, the debug sidecar is the only way to inspect the running environment. The --pid=container: flag requires the target to be running, which is why the target must be kept alive with a long-running command or a debug entrypoint if the original command exits immediately.

Complete Example Session

# ============================================
# PART 1: IDENTIFY THE CRASHED CONTAINER
# ============================================
docker ps -a --filter "name=my-distroless"
# STATUS column shows "Exited (1)" or "Exited (137)"


# ============================================
# PART 2: INSPECT EXIT CODE AND LOGS
# ============================================
docker inspect --format='{{.State.ExitCode}}' my-distroless
# Output: 1 (application error) or 137 (OOMKilled)

docker logs my-distroless
# May be empty if the app buffered output before crashing


# ============================================
# PART 3: TRY THE STANDARD EXEC (EXPECTED TO FAIL)
# ============================================
docker exec -it my-distroless /bin/sh
# Output: OCI runtime exec failed: exec: "/bin/sh": stat /bin/sh: no such file or directory


# ============================================
# PART 4: RUN THE TARGET WITH A KEEP-ALIVE COMMAND
# ============================================
# Start a new container from the same image
# Override the entrypoint to keep it running
docker run -d --name debug-target my-distroless-image tail -f /dev/null
# For distroless images without tail, this may fail; use the original image
# with a long-running command if available, or skip to sidecar attach


# ============================================
# PART 5: ATTACH A DEBUG SIDECAR (DOCKER)
# ============================================
TARGET_ID=$(docker ps -q --filter "name=debug-target")
docker run -it --rm \
  --pid=container:$TARGET_ID \
  --net=container:$TARGET_ID \
  --cap-add=SYS_ADMIN \
  nicolaka/netshoot


# ============================================
# PART 6: INSPECT FROM INSIDE THE SIDECAR
# ============================================
# Inside the netshoot container:
ps aux                    # Target's processes visible
ss -tlnp                  # Target's listening ports
curl localhost:8080       # Test target's service
strace -p 1               # Trace target's PID 1


# ============================================
# PART 7: KUBERNETES EPHEMERAL CONTAINER (DISTROLESS POD)
# ============================================
kubectl debug -it my-distroless-pod \
  --image=busybox \
  --target=app


# ============================================
# PART 8: INSPECT FROM THE EPHEMERAL CONTAINER
# ============================================
# Inside the debug container:
ps aux                    # Target's processes (if namespace shared)
netstat -lntu             # Target's listening ports
cat /proc/1/environ | tr '\0' '\n'  # Target's environment


# ============================================
# PART 9: DEBUG A CRASHED POD WITH --copy-to
# ============================================
kubectl debug -it my-crashed-pod \
  --image=busybox \
  --copy-to=my-debug-pod \
  --share-processes
# A new pod is created; inspect it without affecting the original


# ============================================
# PART 10: CLEAN UP
# ============================================
docker rm -f debug-target
kubectl delete pod my-debug-pod

The ten parts trace the full workflow: inspect the crashed container’s exit code and logs, confirm that exec fails on distroless, start a keep-alive target or use the running target, attach a debug sidecar, inspect from the sidecar, and clean up. Parts 7 and 8 show the Kubernetes equivalent.

Quick Reference

Docker Sidecar Flags

Flag	Purpose
`--pid=container:<id>`	Share target’s PID namespace
`--net=container:<id>`	Share target’s network namespace
`--cap-add=SYS_ADMIN`	Required for `strace`, `tcpdump` on shared processes
`nicolaka/netshoot`	Debug image with network and process tools
`busybox`	Minimal debug image with `sh`

Kubernetes kubectl debug

Flag	Purpose
`--target=<container>`	Share PID namespace of target container
`--image=<image>`	Debug image (busybox, netshoot, etc.)
`--copy-to=<pod>`	Create a copy of the pod for debugging
`--share-processes`	Enable process namespace sharing
`-it`	Interactive terminal

Exit Code Meanings

Code	Meaning
0	Clean exit
1	Application error
137	SIGKILL (often OOMKilled)
143	SIGTERM (normal stop)

Debug Image Choice

Image	Tools	Use When
`busybox`	`sh`, `ls`, `ps`, `cat`	Minimal inspection
`nicolaka/netshoot`	`curl`, `dig`, `tcpdump`, `ss`, `strace`	Network and process debugging
`alpine`	`sh`, `apk`	Need to install packages

Best Practices

✅ Do This:

# Use --pid=container: to see target processes
docker run -it --rm --pid=container:$ID --net=container:$ID nicolaka/netshoot  # ✅

# Use kubectl debug --target for distroless pods
kubectl debug -it my-pod --image=busybox --target=app                  # ✅

# Inspect exit code and logs before attaching
docker inspect --format='{{.State.ExitCode}}' my-container             # ✅

# Clean up debug pods created with --copy-to
kubectl delete pod my-debug-pod                                        # ✅

❌ Don’t Do This:

# Expect docker exec to work on distroless images
docker exec -it my-distroless /bin/sh  # fails: no shell              # ❌

# Attach a debug container without sharing namespaces
docker run -it busybox  # sees its own network, not the target's        # ❌

# Assume kubectl debug --target always shares PID namespace
# Check ps output; if target processes are missing, use --share-processes  # ❌

# Modify the production image to add debugging tools
# Increases attack surface; use ephemeral containers instead             # ❌

Common Pitfalls

Pitfall	Why It Happens	Fix
`ps aux` shows only debug container	PID namespace not shared	Use `--pid=container:` or `--target`
`curl localhost` tests wrong service	Network namespace not shared	Use `--net=container:` or `--target`
`strace` permission denied	Missing `SYS_ADMIN` capability	Add `--cap-add=SYS_ADMIN`
Target processes not visible in kubectl debug	Runtime does not support PID sharing	Enable `shareProcessNamespace: true`
Debug pod remains after session	`--copy-to` creates a real pod	Delete it manually

Real-World Examples

1. Docker Sidecar with Netshoot

docker run -it --rm --pid=container:$ID --net=container:$ID nicolaka/netshoot

2. Docker Sidecar with Busybox

docker run -it --rm --pid=container:$ID --net=container:$ID busybox

3. Kubernetes Ephemeral Container

kubectl debug -it my-pod --image=busybox --target=app

4. Kubernetes Copy for Crashed Pod

kubectl debug -it my-pod --image=busybox --copy-to=debug-pod

5. Check Exit Code

docker inspect --format='{{.State.ExitCode}}' my-container

6. View Logs Before Debugging

docker logs my-container

7. Inspect Target Environment

cat /proc/1/environ | tr '\0' '\n'

8. Access Target Filesystem

ls /proc/1/root/

9. Trace Target Process

strace -p 1  # inside debug container with shared PID namespace

10. Clean Up

docker rm -f debug-target
kubectl delete pod debug-pod

Visual

Docker Sidecar Namespace Sharing

┌──────────────────────────────────────────────────────────────┐
│  TARGET CONTAINER (distroless)                               │
│  ┌─────────────────────────────────────────┐                │
│  │ PID namespace: app process (PID 1)      │                │
│  │ Network namespace: eth0, port 8080      │                │
│  │ Filesystem: distroless (no shell)       │                │
│  └─────────────────────────────────────────┘                │
│           ▲                                                  │
│           │ --pid=container:                                 │
│           │ --net=container:                                 │
│           │                                                  │
│  ┌────────┴────────────────────────────────┐                │
│  │ DEBUG SIDECAR (netshoot)                │                │
│  │                                         │                │
│  │ Shares target's PID namespace           │                │
│  │ Shares target's network namespace       │                │
│  │ Has its own filesystem (tools)          │                │
│  │                                         │                │
│  │ ps aux → sees target's PID 1            │                │
│  │ curl localhost:8080 → target's service  │                │
│  │ strace -p 1 → traces target's process   │                │
│  └─────────────────────────────────────────┘                │
│                                                              │
│  The target is unchanged. The debug container is separate.   │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Kubernetes Ephemeral Container

┌──────────────────────────────────────────────────────────────┐
│  POD                                                         │
│                                                              │
│  ┌─────────────────┐      ┌─────────────────┐               │
│  │ APP CONTAINER   │      │ EPHEMERAL       │               │
│  │ (distroless)    │      │ CONTAINER       │               │
│  │                 │      │ (busybox)       │               │
│  │ No shell        │      │                 │               │
│  │ App running     │      │ sh available    │               │
│  └────────┬────────┘      └────────┬────────┘               │
│           │                        │                         │
│           └──────────┬─────────────┘                         │
│                      │                                       │
│                      ▼                                       │
│           ┌─────────────────────┐                            │
│           │ SHARED NAMESPACES   │                            │
│           │ (network, PID via   │                            │
│           │  --target)          │                            │
│           └─────────────────────┘                            │
│                                                              │
│  kubectl debug -it pod --image=busybox --target=app          │
│  The ephemeral container is not in the pod spec.             │
│  It runs until the session ends, then stops.                 │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Crashed Container Workflow

┌──────────────────────────────────────────────────────────────┐
│  CONTAINER CRASHES                                           │
│                                                              │
│  ┌─────────────────┐                                        │
│  │ docker ps -a    │  Check STATUS: Exited (1) or (137)    │
│  └────────┬────────┘                                        │
│           ▼                                                  │
│  ┌─────────────────┐                                        │
│  │ docker logs     │  Check application output              │
│  └────────┬────────┘                                        │
│           ▼                                                  │
│  ┌─────────────────┐                                        │
│  │ docker inspect  │  Exit code, mounts, config            │
│  └────────┬────────┘                                        │
│           ▼                                                  │
│  ┌─────────────────┐                                        │
│  │ Start new       │  Same image, keep-alive command        │
│  │ container       │  (or use running target)              │
│  └────────┬────────┘                                        │
│           ▼                                                  │
│  ┌─────────────────┐                                        │
│  │ Attach debug    │  --pid=container: --net=container:     │
│  │ sidecar         │                                        │
│  └─────────────────┘                                        │
│                                                              │
│  If the image has no keep-alive command, use Kubernetes      │
│  --copy-to to create a debuggable copy.                      │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Exit Code Decision

┌──────────────────────────────────────────────────────────────┐
│  EXIT CODE                                                   │
│                                                              │
│  ┌─────────────────┐                                        │
│  │ 0               │  Clean exit; container was expected    │
│  │                 │  to keep running but didn't            │
│  └─────────────────┘                                        │
│                                                              │
│  ┌─────────────────┐                                        │
│  │ 1               │  Application error; check logs         │
│  └─────────────────┘                                        │
│                                                              │
│  ┌─────────────────┐                                        │
│  │ 137             │  SIGKILL; likely OOMKilled             │
│  │                 │  Check memory limits and usage         │
│  └─────────────────┘                                        │
│                                                              │
│  ┌─────────────────┐                                        │
│  │ 143             │  SIGTERM; normal stop signal           │
│  └─────────────────┘                                        │
│                                                              │
│  The exit code narrows the failure class before              │
│  attaching a debug container.                               │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Summary

Item	Value
Distroless problem	No shell, no tools; `exec` fails
Docker sidecar flags	`--pid=container:`, `--net=container:`
Kubernetes flag	`--target=<container>`
Debug image (network)	`nicolaka/netshoot`
Debug image (minimal)	`busybox`
Crashed container first step	Inspect exit code and logs
Crashed pod debugging	`--copy-to` creates a debuggable copy
Namespace sharing	Required to see target’s processes and network

Key takeaways:

Distroless images cannot be debugged with exec. The absence of a shell is intentional. The ephemeral container pattern attaches a separate container with tools, sharing the target’s namespaces, without modifying the target image .
Namespace sharing is what makes the debug container useful. Without --pid=container: (Docker) or --target (Kubernetes), the debug container sees its own processes and network, not the target’s. The ps and curl commands must operate on the target’s namespaces to be meaningful .
A crashed container requires inspection before attachment. The exit code (1, 137, 143) and the logs are the first sources of information. If the container has exited, exec is impossible; the approach is to start a new container from the same image with a keep-alive command, or use --copy-to in Kubernetes .
Debug images should match the task. busybox provides a shell and basic utilities for filesystem inspection. nicolaka/netshoot provides network tools (curl, dig, tcpdump) and process tools (strace, ss) for deeper diagnosis. The choice depends on whether the problem is in the application logic or the network layer .
Ephemeral containers are temporary by design. In Kubernetes, they cannot be removed from the pod spec; they stop when the session ends. In Docker, the --rm flag removes the debug container when it exits. Copied pods (--copy-to) are real pods and must be deleted manually .

Remember: Debugging a distroless container is not about getting inside it. It is about attaching a second container that shares its namespaces and has the tools the first one lacks. The target container remains unchanged, the production image remains minimal, and the debug container exists only for the duration of the investigation. The pattern works for running containers and, with the --copy-to variant, for crashed pods. The exit code and logs are always the first step; the ephemeral container is the second step, used when the first step does not explain the failure.

Was this helpful?

Stop using slow, ad-bloated tool sites! 🤮

🔎 Search “KandZ Tools” on Google to use many professional utilities for free.

KandZ.me is the ultimate minimalist hub for:
✅ Finance (Mortgage, Interest, Inflation)
✅ Tech (Base64, JSON, Dev Suite, IP)
✅ Health (BMI, BMR, TDEE)
✅ Productivity (Timer, Workspace, QR)

⚡️ Fast & Private
🔒 No data leaves your device
💎 100% Free

🔗 Use it now: https://tools.kandz.me
🔖 Bookmark it—you’ll need it later!