🔏 Sigstore

What is Sigstore?

Sigstore brings keyless signing and verification to software supply chains. Instead of managing long-lived GPG keys, you get short-lived certificates tied to your identity provider. Sign an artifact, the signature goes into an append-only transparency log, and anyone can verify the provenance later.

The stack has a few moving parts: Fulcio issues short-lived certificates based on OIDC identity, Rekor stores signatures in an immutable transparency log, CTLog provides certificate transparency, TUF distributes the root of trust, TSA handles RFC 3161 timestamps, and Cosign ties it all together for signing and verifying container images, binaries, and SBOMs.

If you care about knowing who built what and when in your CI/CD pipeline, Sigstore is the answer.

What We Built

We deployed a full internal Sigstore instance for our developer experience platform. The setup runs on OpenShift, uses GitLab as the OIDC provider, and covers the complete stack — all 7 components: Fulcio, Rekor, CTLog, Trillian (the Merkle tree backend with MySQL), TUF, TSA, and Redis for Rekor caching.

The deployment is based on the upstream Sigstore scaffold Helm chart, wrapped in our own chart with environment-specific values for review, acceptance, and production. CI/CD runs through GitLab pipelines using a generic Helm deploy module.

Once it's up, signing a container image is as simple as:

cosign sign <image>

Cosign handles the OIDC dance with GitLab, requests a certificate from Fulcio, signs the image, uploads the signature to Rekor, and gets a timestamp from TSA. Verification is equally straightforward:

cosign verify <image> \
  --certificate-identity=<email> \
  --certificate-oidc-issuer=https://your-gitlab.example.com

That's the dream. And when it works, it's beautiful.

Where It Gets Painful

Here's the thing about Sigstore: the upstream Helm chart and docs are built for getting started. Cloud-first, public OIDC, experimental installs. That's fine for a demo. But running it as production infrastructure inside a corporate network? You're on your own for most of the hard parts.

The Helm Chart Is Geared for Demos

Single replicas, weak resource limits, ephemeral storage assumptions. We had to override security contexts for OpenShift UID assignment, configure internal registry mirrors for every single image, and tune resource limits for quota compliance. Even with a focused values file, there's a lot of surface area to get right.

Certificate Rotation Will Bite You

This was our worst issue. The Fulcio createcerts job runs on every deployment when enabled. It generates a new root CA each time. But CTLog has a protection mechanism that prevents overwriting its trusted roots — so after a Helm upgrade, Fulcio signs with a new CA that CTLog doesn't trust.

The error looks like this:

POST /api/v1/signingCert returned 500: 
"Error entering certificate in CTL"

What actually happened: Fulcio has a new root CA from March 3rd, CTLog still trusts the one from February 27th. Certificate verification fails silently.

We wrote a fix script that diagnoses the mismatch, patches the CTLog secret with the current Fulcio root, and restarts CTLog. It's idempotent and safe to run multiple times. But the fact that we needed it at all tells you something about the operational maturity of the chart.

The long-term fix: make the createcerts job only run on initial install, never on upgrades. We considered Helm hooks (pre-install) but those don't work here — the jobs depend on other chart resources that don't exist during the pre-install phase. Instead, the jobs run as normal Kubernetes Jobs with force: false so they skip if certs already exist.

TUF Root Breaks After Tree Regeneration

Another fun one. Helm upgrades re-ran the Rekor createtree job, which created a new Trillian tree with a new signing key pair. Now the TUF-distributed trusted_root.json contains the old Rekor public key, but Rekor is serving a new one.

Cosign successfully uploads signatures but then fails to verify them:

failed to verify log inclusion: 
not enough verified log entries from transparency log: 0 < 1

The fix: regenerate and redeploy the TUF root with the current Rekor public key. The prevention: same as above — initialization jobs must be idempotent and skip on upgrades.

TUF trusted_root.json Was Incomplete

Our TUF repository shipped with certificate_authorities: null in the trusted root. That meant cosign initialize worked, but cosign couldn't use TUF for automatic service discovery. Users had to specify every URL manually with flags or environment variables. Not exactly the seamless experience Sigstore promises.

What Actually Works — Our Playbook

Fork and harden the chart. We wrapped the upstream scaffold chart in our own, with environment-specific values for review, AT, and production. All images point to internal registry mirrors. Resource limits, security contexts, and routes are configured for OpenShift.

Make initialization jobs idempotent. The createcerts and createtree jobs check if resources already exist before creating them. force: false everywhere. No Helm hooks — let them run as normal Jobs after all resources are deployed.

Test the full flow in CI. Our system tests hit every endpoint — Fulcio root cert, Rekor public key, CTLog, TUF root metadata, TSA cert chain. If any of them return unexpected results after a deploy, we know immediately.

Document the gotchas. We maintain docs for every issue we've hit — certificate rotation, TUF root mismatches, the incomplete trusted_root.json. Future us will thank past us.

The Rekor Situation

One more thing worth mentioning: Rekor v1 — the version that ships with the scaffold chart — is in maintenance mode. The Sigstore team is focusing on Rekor v2 (also called rekor-tiles), which redesigns the transparency log on top of a tiled Merkle tree structure. It's a significant architectural improvement.

Rekor v2 already has its own Helm chart. But it hasn't been integrated into the scaffold chart yet. So if you're deploying Sigstore today via the scaffold, you're getting Rekor v1 — and you should know that active development has moved on.

For us, that means we're running a component that's getting security fixes but no new features. At some point we'll need to migrate to rekor-tiles, and that migration path isn't documented yet either. Another thing to keep an eye on.

Closing Thoughts

Sigstore is a transformational project for supply-chain security. The idea of keyless, identity-based signing with transparency logs is exactly right. But the gap between "it works in a demo" and "it runs reliably in production" is significant.

The upstream chart assumes you'll figure out operations yourself. Certificate lifecycle, job idempotency, TUF root management, internal OIDC integration — all of that is on you. If you're planning to adopt Sigstore internally, budget real engineering time for hardening. It's worth it — the auditability and trust you get across your delivery pipeline is genuinely valuable. Just don't expect it to be plug-and-play.

Help Wanted

If you've deployed Sigstore internally and solved problems we haven't — or if you're struggling with the same issues — I'd love to hear from you. Drop a comment below or reach out on GitHub. This stuff is hard, and the more we share operational knowledge, the better it gets for everyone.