Turning the BOLA Test Matrix into Automation#

The IDOR test plan defined a matrix: same tenant with a different role, different tenant, expired membership, write attempts with read-only access. As long as that matrix lives in a document, it stops working the moment the product grows. The person adding a new endpoint will not read the document. The matrix has to live in code.

The idea is simple: keep the authorization combinations as a fixed list in a test file, and let no endpoint pass CI without going through it.

Why a document matrix dies: the endpoint-copy problem#

A matrix in a document is correct on day one and stale by month two.

The cause is not discipline, it is how endpoints multiply.

The main resource endpoint stays protected, its copies do not.

A concrete scenario unfolds like this: GET /api/invoices/:id gets an authorization check when first written.

The code calls requireInvoiceAccess, adds a session.tenantId filter to the db.invoice query, and passes review.

A few sprints later, three side paths appear.

The first is PDF export: GET /api/invoices/:id/pdf.

The product asks for a "Download PDF" button, and the developer copies the existing handler into a new file wired to a PDF library.

The authorization line gets lost in the copy because the new file starts from the library sample, not from the old handler.

The second is webhook retry: POST /api/invoices/:id/retry.

Notifications from the payment provider drop occasionally, so support asks for a "resend" button.

This endpoint returns no invoice, it only enqueues a job.

The author assumes there is nothing to protect since no data is returned.

But the queued job carries the invoice total and customer details in full.

The third is admin preview: GET /api/admin/invoices/preview?invoiceId=....

Support wants to see a customer invoice, so a quick internal screen is built.

The ID from the query string goes straight into db.invoice.findUnique.

Because it sits under the admin router, everyone assumes the admin panel session is enough, yet a panel session and object ownership are different things.

All three share one trait: none is written with bad intent, each is born in a separate file under time pressure.

A document matrix cannot catch them because nobody runs the matrix.

The matrix survives only when a test next to each new route file runs it.

That is why the factory, client, runner, and CI registry below are built as one unit.

A test-user factory#

The matrix runs on four users with different relationships to the target object:

These four are created from scratch by a seed script on every test run. An IDOR test with two hand-made users runs once; a factory runs on every pull request.

The factory does more than open four records.

It sets up two isolated tenants, writes roles into the membership table, places the target object inside the victim tenant, and removes everything when the run ends.

Without cleanup, data from one run breaks the expectations of the next.

So the factory below ships seedMatrixWorld and cleanupMatrixWorld together.

Copy code
import { db } from './db'
import { createSessionToken } from './auth/sessions'

export type MatrixRole = 'owner' | 'member' | 'outsider' | 'expired'

export type MatrixWorld = {
  tenantVictim: { id: string }
  tenantOther: { id: string }
  users: Record<MatrixRole, { id: string; email: string; token: string; tenantId: string }>
  invoiceId: string
  victimMarker: string
}

const runTag = `mx-${Date.now().toString(36)}-${Math.floor(Math.random() * 1e6)}`

export async function seedMatrixWorld(): Promise<MatrixWorld> {
  await cleanupMatrixWorld(runTag)

  const tenantVictim = await db.tenant.create({
    data: { name: `victim-${runTag}` },
  })
  const tenantOther = await db.tenant.create({
    data: { name: `other-${runTag}` },
  })

  const victimMarker = `victim-note-${runTag}`

  const owner = await db.user.create({
    data: { email: `owner-${runTag}@test.local`, tenantId: tenantVictim.id },
  })
  await db.membership.create({
    data: { userId: owner.id, tenantId: tenantVictim.id, role: 'owner', revokedAt: null },
  })

  const member = await db.user.create({
    data: { email: `member-${runTag}@test.local`, tenantId: tenantVictim.id },
  })
  await db.membership.create({
    data: { userId: member.id, tenantId: tenantVictim.id, role: 'member', revokedAt: null },
  })

  const outsider = await db.user.create({
    data: { email: `outsider-${runTag}@test.local`, tenantId: tenantOther.id },
  })
  await db.membership.create({
    data: { userId: outsider.id, tenantId: tenantOther.id, role: 'member', revokedAt: null },
  })

  const expired = await db.user.create({
    data: { email: `expired-${runTag}@test.local`, tenantId: tenantVictim.id },
  })
  await db.membership.create({
    data: {
      userId: expired.id,
      tenantId: tenantVictim.id,
      role: 'member',
      revokedAt: new Date(),
    },
  })

  const invoice = await db.invoice.create({
    data: {
      tenantId: tenantVictim.id,
      createdById: owner.id,
      total: 1250,
      notes: victimMarker,
      lines: { create: [{ label: victimMarker, amount: 1250 }] },
    },
  })

  const sessionFor = async (userId: string, tenantId: string) =>
    createSessionToken({ userId, tenantId })

  return {
    tenantVictim: { id: tenantVictim.id },
    tenantOther: { id: tenantOther.id },
    users: {
      owner: { id: owner.id, email: owner.email, tenantId: tenantVictim.id, token: await sessionFor(owner.id, tenantVictim.id) },
      member: { id: member.id, email: member.email, tenantId: tenantVictim.id, token: await sessionFor(member.id, tenantVictim.id) },
      outsider: { id: outsider.id, email: outsider.email, tenantId: tenantOther.id, token: await sessionFor(outsider.id, tenantOther.id) },
      expired: { id: expired.id, email: expired.email, tenantId: tenantVictim.id, token: await sessionFor(expired.id, tenantVictim.id) },
    },
    invoiceId: invoice.id,
    victimMarker,
  }
}

export async function cleanupMatrixWorld(tag: string = runTag): Promise<void> {
  await db.invoiceLine.deleteMany({ where: { label: { contains: tag } } })
  await db.invoice.deleteMany({ where: { notes: { contains: tag } } })
  await db.membership.deleteMany({ where: { user: { email: { contains: tag } } } })
  await db.user.deleteMany({ where: { email: { contains: tag } } })
  await db.tenant.deleteMany({ where: { name: { contains: tag } } })
}

Three details in this code carry the accuracy of the matrix.

First, runTag is unique per run, so parallel runs never see each other invoices.

Second, the expired user gets a live token on purpose while the membership row carries revokedAt; what is tested is membership state, not token shape.

Third, victimMarker is written into both the invoice note and the line description, so the runner catches a leak through any view.

API test client: token and tenant header#

The matrix is not written with raw fetch calls.

Every request must carry the same two headers: the user token and the tenant identity.

Written by hand on each row, one row will eventually miss a header and produce a false pass.

So a thin client sits on top of the factory.

Copy code
export type HttpMethod = 'GET' | 'POST' | 'PUT' | 'PATCH' | 'DELETE'

export type ApiResponse = {
  status: number
  bodyText: string
  bodyJson: unknown
}

const baseUrl = process.env.TEST_BASE_URL ?? 'http://localhost:3000'

export async function apiAs(
  world: MatrixWorld,
  role: MatrixRole,
  method: HttpMethod,
  path: string,
  body?: unknown,
): Promise<ApiResponse> {
  const user = world.users[role]
  const res = await fetch(`${baseUrl}${path}`, {
    method,
    headers: {
      'content-type': 'application/json',
      authorization: `Bearer ${user.token}`,
      'x-tenant-id': user.tenantId,
    },
    body: body === undefined ? undefined : JSON.stringify(body),
  })
  const bodyText = await res.text()
  let bodyJson: unknown = null
  try {
    bodyJson = bodyText ? JSON.parse(bodyText) : null
  } catch {
    bodyJson = null
  }
  return { status: res.status, bodyText, bodyJson }
}

export function invoicePaths(invoiceId: string) {
  return {
    read: `/api/invoices/${invoiceId}`,
    write: `/api/invoices/${invoiceId}`,
    pdf: `/api/invoices/${invoiceId}/pdf`,
    retry: `/api/invoices/${invoiceId}/retry`,
    preview: `/api/admin/invoices/preview?invoiceId=${invoiceId}`,
    remove: `/api/invoices/${invoiceId}`,
  }
}

The client stays thin by design.

Token creation belongs to the factory, the client only carries the values.

The x-tenant-id header always comes from the acting user own tenant; scenarios that tamper with the header get their own rows.

bodyText is kept as raw text because the leak check runs before JSON parsing; some side paths return PDF or HTML.

The invoicePaths helper makes every runner row use the same ID.

The matrix skeleton#

Copy code
const matrix = [
  { user: 'owner', action: 'read', expect: 200 },
  { user: 'member', action: 'read', expect: [403, 404] },
  { user: 'outsider', action: 'read', expect: [403, 404] },
  { user: 'expired', action: 'read', expect: [401, 403, 404] },
  { user: 'member', action: 'write', expect: [403, 404] },
]

for (const row of matrix) {
  const res = await api(row.user, row.action, targetObjectId)
  expect(row.expect).toContain(res.status)
  if (res.status === 200) {
    expect(res.body).not.toContain(victimMarker)
  }
}

The key decision here is the oracle: an unauthorized request must not return 200, and the body must not contain a marker unique to the victim's data (victimMarker). The test does not dictate 403 versus 404; both are accepted because from the outside they mean the same thing: no access.

The skeleton starts with five rows, but that is not enough for production protection.

The full table below grows the skeleton to ten rows and treats side paths with the same weight as the main path.

The full ten-row matrix#

The table shows the role-by-action intersection at a glance.

Ten rows are not arbitrary: four roles crossed with five action types (read, write, admin-copy, export, soft-delete) set the lower bound of the matrix.

Rows 1 and 2 are positive controls.

Without them, the matrix would also pass a broken layer that denies everything.

Rows 3 and 4 check role separation inside the same tenant; the member-can-read but cannot-write rule is verified here.

Rows 5 through 8 try four different doors from an outside tenant.

Export, admin preview, and soft-delete get separate rows because nothing guarantees they call the same requireInvoiceAccess helper behind the scenes.

Rows 9 and 10 catch the revoked membership.

401 is accepted for expired rows only, because some session layers invalidate the token once membership drops.

When a new action type enters the table, adding a row is not enough; the request builder in the runner gets a branch with the same name.

The matrix runner#

The runner is the loop that executes the table.

It does three things: send the request with the right identity, compare the status against the expected set, and scan the body for the leak marker on any 200.

Copy code
import { afterAll, beforeAll, describe, expect, it } from 'vitest'
import { cleanupMatrixWorld, seedMatrixWorld, type MatrixWorld } from './matrix-factory'
import { apiAs, invoicePaths, type HttpMethod, type MatrixRole } from './matrix-client'

type MatrixRow = {
  name: string
  role: MatrixRole
  method: HttpMethod
  path: (paths: ReturnType<typeof invoicePaths>) => string
  body?: unknown
  expect: number[]
}

function buildRows(invoiceId: string): MatrixRow[] {
  const paths = invoicePaths(invoiceId)
  return [
    { name: 'owner read', role: 'owner', method: 'GET', path: () => paths.read, expect: [200] },
    { name: 'owner write', role: 'owner', method: 'PUT', path: () => paths.write, body: { notes: 'owner edit' }, expect: [200] },
    { name: 'member read denied', role: 'member', method: 'GET', path: () => paths.read, expect: [403, 404] },
    { name: 'member write denied', role: 'member', method: 'PUT', path: () => paths.write, body: { notes: 'x' }, expect: [403, 404] },
    { name: 'outsider read denied', role: 'outsider', method: 'GET', path: () => paths.read, expect: [403, 404] },
    { name: 'outsider export denied', role: 'outsider', method: 'GET', path: () => paths.pdf, expect: [403, 404] },
    { name: 'outsider admin preview denied', role: 'outsider', method: 'GET', path: () => paths.preview, expect: [403, 404] },
    { name: 'outsider delete denied', role: 'outsider', method: 'DELETE', path: () => paths.remove, expect: [403, 404] },
    { name: 'expired read denied', role: 'expired', method: 'GET', path: () => paths.read, expect: [401, 403, 404] },
    { name: 'expired retry denied', role: 'expired', method: 'POST', path: () => paths.retry, body: {}, expect: [401, 403, 404] },
  ]
}

describe('invoice BOLA matrix', () => {
  let world: MatrixWorld

  beforeAll(async () => {
    world = await seedMatrixWorld()
  })

  afterAll(async () => {
    await cleanupMatrixWorld()
  })

  for (const row of buildRows(':id')) {
    it(row.name, async () => {
      const paths = invoicePaths(world.invoiceId)
      const res = await apiAs(world, row.role, row.method, row.path(paths), row.body)
      expect(row.expect).toContain(res.status)
      if (res.status === 200) {
        expect(res.bodyText).not.toContain(world.victimMarker)
      }
    })
  }
})

The loop variable moves into the test title, so the CI output names the broken row directly.

Status comparison uses toContain; no single code is forced.

Every row that returns 200 gets a body scan, including rows expected to pass, which also catches the owner row returning another tenant data by mistake.

The beforeAll and afterAll pair wires the factory setup and cleanup into the test.

Oracle depth: 403 versus 404, marker placement, no timing#

The oracle is the rule that decides pass or fail.

Two layers are used here: the status set and the body marker.

403 and 404 are both accepted.

The reason is the product ID-hiding choice.

Some teams return 403 for a forbidden object, others return 404; from the outside both say no access.

If the test forces one, the matrix turns red the day the team switches to 404 for enumeration protection, and nobody takes the failure seriously.

401 is accepted on expired rows only.

A session layer may treat a dropped membership as an invalid token, and then 401 is the right answer.

On active memberships 401 is rejected; a valid token returning 401 points at a session problem, not an authorization problem.

victimMarker placement is the second call that matters.

The marker goes into a field returned by every view.

That is why the factory above writes it into both notes and the line label.

A list view may truncate notes, a PDF may pull lines separately; with the marker in two places, one view difference is not enough to miss it.

The marker is unique per run, never a fixed string.

A fixed string collides across parallel runs and repeated runs with other tests data.

Timing is not used as an oracle.

Inferring "record exists but forbidden" from response time is not stable in CI.

Runners under load produce shaky durations, and the test turns flaky.

A flaky test ends with the team muting it.

The logic is plain: status plus body marker give the same answer on every run, timing does not.

CI integration: coverage registry and forced runs#

A matrix that does not run in CI is no different from a document.

Two parts are needed: a registry pattern and a workflow.

The registry checks at build time that every endpoint entered the matrix.

Copy code
type FixtureRef = { invoiceId: string; victimMarker: string }

type CoveredRoute = {
  route: string
  run: (fixture: FixtureRef) => Promise<void>
}

const coveredRoutes: CoveredRoute[] = []

export function coverEndpoint(route: string, run: (fixture: FixtureRef) => Promise<void>) {
  coveredRoutes.push({ route, run })
}

export function coveredRouteNames(): string[] {
  return coveredRoutes.map((r) => r.route)
}

coverEndpoint('GET /api/invoices/:id', async () => {})
coverEndpoint('GET /api/invoices/:id/pdf', async () => {})
coverEndpoint('POST /api/invoices/:id/retry', async () => {})
coverEndpoint('GET /api/admin/invoices/preview', async () => {})
coverEndpoint('DELETE /api/invoices/:id', async () => {})

The registry alone is not enough; a test compares the route list against the registry list.

Copy code
import { expect, it } from 'vitest'
import { appRoutes } from './app-routes'
import { coveredRouteNames } from './matrix-registry'

it('every invoice route has matrix coverage', () => {
  const missing = appRoutes.filter((r) => !coveredRouteNames().includes(r))
  expect(missing).toEqual([])
})

When someone adds an endpoint without a registry line, this test turns red.

So the forgotten PDF path gets caught at pull request time.

The workflow file runs the matrix test on every pull request:

Copy code
name: bola-matrix
on:
  pull_request:
  push:
    branches: [main]
jobs:
  matrix:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: 'npm'
      - run: npm ci
      - run: npm run db:migrate:test
      - run: npm run test:matrix -- --run

Database migration for the test database is a separate step in the workflow.

The matrix never runs against the development database; the connection string points at the test database.

The test:matrix command runs only matrix files, so the result reads fast.

Coverage and behavior checks run in the same job; one guards the list, the other guards the behavior.

Five common mistakes#

The first mistake: testing only the happy path.

Owner getting 200 is read as a matrix pass.

But BOLA is about denial logic; without denial rows the matrix is empty.

The second mistake: counting an empty 200 as a pass.

Some authorization layers let the request through but strip the body.

The status reads 200, the test turns green, yet the decision is wrong.

That is why the runner keeps the marker check and why positive rows also verify the body.

The third mistake: testing with a single tenant.

With one tenant there is no outsider row and the test stays blind.

Two tenants are a required part of the factory, not an option.

The fourth mistake: testing with a scope-poor token.

When token creation and request headers carry different tenants, denial arrives for the wrong reason.

The factory mints the token and the header from the same source, so a denial points at membership.

The fifth mistake: leaving the matrix in documents.

A matrix on a wiki page never gains the new endpoint.

The registry pattern and the coverage test make the list part of the code.

Limitations#

This matrix does not catch everything.

Rate limits sit outside the test.

Once the per-minute threshold trips, the API returns 429, which the matrix could misread as denial.

So matrix runs use a test key exempt from rate limits, or run with low parallelism.

Parallel-run isolation is the second limit.

Two matrix runs against the same database at the same time can clash on tags.

The runTag reduces this, but full safety comes from each CI job using its own database schema.

The third limit is the data environment.

The matrix runs against the test database only, never against production data.

A revoked-membership scenario run in production touches real user tokens and leaves hard-to-reverse dirt.

The fourth limit is business-flow permission.

The matrix tests object ownership, not workflow rules such as an approval chain.

Rules like "a member may read an invoice but may not approve it" need their own tests.

Short result#

Closing BOLA is not fixing one endpoint; it is making the matrix run against every new endpoint automatically. Four users, one expectation table, one leak-marker check. While those three run in CI, the authorization decision belongs to code, not to people remembering to check.

The remaining work is small: add a registry line when a route opens, generate the ten runner rows for that route, and when CI shows a red row, inspect the permission helper and the session.tenantId filter.