A PostgreSQL 16 cluster on a Proxmox host came back from a UPS failure and the primary refuses to start — the log shows “invalid page in block 4823 of relation base/16384/12345” and PostgreSQL is refusing to serve any query that would touch the affected table. A managed service provider running PostgreSQL on ZFS lost a mirror leg during resilver and the pg_wal directory is now returning read errors on segments that PostgreSQL needs for crash recovery. A small research lab’s single-node PostgreSQL that has been running fine for eight years came back from a filesystem crash with the pg_control file zeroed and every attempt to start refusing with “the database system was shut down at …” followed by a hex address that is clearly garbage.

We recover PostgreSQL clusters when the storage under them fails. PostgreSQL is well-designed for surviving software crashes and interrupted transactions, but it is not designed to survive its storage lying about what happened. When RAID fails mid-write, when a disk returns stale blocks after a resilver, when a filesystem returns a page that no longer belongs to the file it thinks it does, PostgreSQL will either refuse to start or start and then produce results that no longer correspond to committed data. That is where we come in.

Heap pages, WAL, and what breaks when storage lies

PostgreSQL stores tables in heap files under base/<dbid>/<relid>, each file broken into 8 KB pages. Every page begins with a 24-byte PageHeaderData carrying pd_lsn, pd_checksum (when enabled), and pointers pd_lower and pd_upper delimiting the used and free portions. An ItemIdData array grows downward from pd_lower; tuples grow upward from pd_upper. Special space at the end of the page is used by index pages and left empty by heap pages.

Every change to a page — INSERT, UPDATE, DELETE, index update, transaction commit — is written to the write-ahead log (WAL) under pg_wal/ before the heap page itself is flushed to disk. This is what makes PostgreSQL crash-safe: on startup after an unclean shutdown, the WAL is replayed forward from the last checkpoint LSN, and any page whose on-disk pd_lsn is behind the WAL is refreshed from the log. The system reaches a consistent state before accepting queries.

Storage-level corruption breaks this design in specific ways. When RAID drops mid-write and returns a page whose pd_lower or pd_upper values point outside the page, PostgreSQL raises “invalid page in block.” When an SSD returns a page whose contents are stale from a previous erase block, the page checksum may still pass but the pd_lsn will be in the past relative to the WAL, and replaying WAL forward will fail because the base state is wrong. When pg_wal itself is on failing storage and segments are missing or torn, WAL replay stops mid-recovery and the cluster comes up in a partially-consistent state, or refuses to come up at all.

Error signatures that indicate storage problems

• invalid page in block X of relation base/Y/Z — the page’s internal structure is inconsistent. pd_lower is greater than pd_upper, or an ItemIdData points outside the page, or the special space is malformed. Reliable indicator of a torn or corrupt page from storage.
• page verification failed, calculated checksum X but expected Y — data_checksums was enabled at initdb time and this page’s stored checksum does not match its contents. Directly attributable to storage returning a page whose bytes have changed since PostgreSQL wrote it.
• could not read block X in file “base/Y/Z”: read only 0 of 8192 bytes — the filesystem returned a short read, usually indicating a truncated or damaged file. Common after a filesystem crash where the relation’s inode was damaged.
• heap tuple with XMAX X but no matching visibility map entry — internal consistency between the heap and the visibility map is broken. Typically indicates a torn write between the two files.
• could not open file “pg_wal/00000001000000000000000X”: No such file or directory during recovery — a WAL segment PostgreSQL needs to reach a consistent state is missing. Recovery stalls; the cluster cannot come up.
• invalid resource manager ID in primary checkpoint record or invalid magic number in log segment — the WAL is torn at the segment level. Recovery cannot proceed past this point.
• pg_control checksum mismatch — the tiny but critical global control file has been corrupted. Without it, the cluster cannot determine the correct recovery LSN. Special reconstruction is required.

How we recover a PostgreSQL cluster

Every disk involved is imaged first, bit-exact, before anything else. When PostgreSQL is on ZFS, ext4, XFS, or a filesystem stacked on RAID or LVM, we image at the block-device level below the filesystem so that even filesystems that have crashed can be reconstructed from a stable snapshot. Nothing writes to the original disks after they arrive.

Filesystem reconstruction comes next. When ZFS is involved — and a growing share of PostgreSQL deployments run on ZFS — we work through the imported pool’s uberblocks to find the newest consistent state and rebuild the dataset. For ext4 or XFS, standard filesystem-level recovery reassembles the data directory. When the underlying storage is a RAID array or LVM volume that failed mid-write, we determine stripe geometry and volume-group layout from the imaged data rather than trusting the failed configuration.

From the reconstructed PGDATA directory, we walk each heap file page by page. Pages whose pd_lower/pd_upper are valid and whose checksums pass (when enabled) are preserved intact. Pages that fail get inspected: often a tuple heap is still walkable even when the ItemIdData array is damaged, and we extract tuples by pattern-matching HeapTupleHeader structures. Where pg_wal survives, we selectively replay WAL forward onto the last consistent base state. Where pg_wal is destroyed but the heap files are stable, we deliver a cluster that starts at the last successful checkpoint — sometimes with a few minutes of the newest committed transactions unavailable, sometimes more.

When pg_control is destroyed, we rebuild it by reading pg_lsn values from the WAL and control-record fragments across the cluster. This is the most involved recovery scenario and requires deep knowledge of the on-disk format for the PostgreSQL major version involved, but it is achievable when the underlying heap files and at least some WAL survives.

Extensions, replicas, and logical replication

PostgreSQL extensions that store data outside the standard heap — PostGIS, TimescaleDB, Citus, pgvector — use the same page format and recover the same way. Streaming replicas that lost their primary and cannot promote because of WAL divergence can often be recovered by imaging the standby, extracting the heap state, and rebuilding on a new primary. Logical replication slots that got stuck do not typically require recovery work — they require draining or reset by a DBA on a working cluster.

What we don’t do

We do not undo committed DELETEs or DROP TABLEs when PostgreSQL wrote them to WAL cleanly and the storage confirmed the flush. The MVCC row versions may still be in the heap for a while, but PostgreSQL will not consider them visible after commit, and reconstructing them requires either running against a pre-VACUUM state (possible, occasionally) or accepting that the data is gone. Point-in-time recovery from base backups plus WAL archives is the correct tool for that; we cannot substitute for a functional backup rotation. We do help when the destructive operation coincided with a hardware failure — a torn WAL flush during the delete, a power event during the drop, a filesystem crash that left the heap and WAL inconsistent.

Related recovery services

See our database data recovery hub, Linux data recovery for the underlying host, RAID data recovery when array failure caused the corruption, and MySQL/MariaDB recovery when both databases co-exist on a shared host.

]]>