Our Hyvä module testing pipeline: 50 migrations taught us this

Every Hyvä-compatible module we ship passes a four-stage testing pipeline before going live in the catalog. We built this pipeline iteratively over 50+ Hyvä migrations and roughly 200 module installations. This post is the architecture of that pipeline, what each stage catches, and what we would build differently if starting over.

Stage 1: the reference store

Every module is first installed against a clean Hyvä reference store. The reference is Magento Open Source 2.4.7 plus Hyvä Themes 1.3 plus Hyvä Checkout. Catalog has 200 products in 6 categories with realistic media weight. No other third-party modules.

The reference store is the cleanest possible environment. If a module breaks here, the issue is the module, not the surrounding code. The reference store catches: install-time errors, layout XML conflicts with Hyvä, missing Composer dependencies, schema migration failures, admin UI rendering bugs.

What the reference store does not catch: third-party module conflicts, real-traffic edge cases, performance regressions caused by interaction effects.

Stage 2: the Lighthouse budget

Stage 2 runs the reference store against Lighthouse mobile audits before and after the module install. The module passes only if mobile Performance stays above 90 with the module active.

Nine specific checks are part of the budget:

1. Total blocking time under 200ms
2. Largest Contentful Paint under 2.5s
3. Cumulative Layout Shift under 0.1
4. No legacy JavaScript (ES5 polyfills for browsers nobody uses)
5. No unused CSS over 20KB
6. Render-blocking resources under 600ms
7. Image formats current (WebP minimum)
8. No render-blocking third-party iframes
9. Server response under 600ms TTFB

Most module ships pass this stage. The ones that fail tend to fail on points 1, 4, or 6. Failed audits return to the developer with the specific Lighthouse trace.

Stage 3: the third-party module conflict matrix

Stage 3 installs the module against the 20 most-popular Magento modules combined. We maintain a matrix of which Hyvä compatibility modules are active in this configuration: cart, checkout, search, payment gateways, analytics, GDPR consent.

The conflict matrix catches: CSS specificity conflicts, JavaScript event-handler stomping (one module overriding another's Alpine bindings), Composer dependency conflicts, database table migration conflicts.

We have caught regressions at this stage that would have shipped to customers undetected: a checkout-tracking module that broke when installed alongside a GDPR consent module because both registered event listeners on the same Magento dispatch hook.

Stage 4: real-store smoke tests

The final stage runs the module on a staging copy of one of our customer stores. Real product catalog, real category structure, real cart and checkout flows. Smoke tests execute the canonical user journeys: home → category → product → add to cart → checkout → order placed.

This stage catches: edge cases specific to real catalogs (configurable products with unusual attribute structures, products with 50+ media items, categories with 5000+ products), real-world performance impact under real cache load, integration-level bugs that only appear with realistic data volumes.

We rotate which customer store hosts stage-4 testing. The owner is informed and compensated for the staging-environment costs. The store never sees production-customer-facing impact because tests run on a staging clone.

What we would build differently

Three things we have learned along the way that we would change if rebuilding from scratch.

Reference store on every Magento minor release. We rebuild the reference store once per major Magento release (2.4.6, 2.4.7) but not on minor patches. We should rebuild on every minor release because patch releases occasionally change behavior in ways that affect module compatibility.

Module versioning against tested-against Magento version. Each module release should declare exactly which Magento patch version it was tested against. We do this loosely in the changelog; we should do it formally in the Composer dependency declaration.

Automated regression for top customer use cases. Stage 4 is manual smoke tests today. Cypress automation against three customer stores would catch regressions faster and reduce the staging-environment burden on those customers.

Why we publish this

The Magento module catalog has too many vendors who ship code with no testing infrastructure. Pages on our blog like the Lighthouse 90 module ship-blocker post explain our policy; this post explains the pipeline that enforces it.

If you are a Magento store evaluating modules, ask your prospective vendor what their testing pipeline looks like. If the answer is hand-wavy, the module probably has not been tested against your specific Magento version, your specific Hyvä variant, or your specific stack of other third-party modules. That is a risk you carry, not them.

Related

• Lighthouse 90 module ship-blocker post covers the policy.
• Magento extension buyer's guide is the framework for buyers.