A MySQL 8 instance backing a busy Magento store came back from a datacenter power event with InnoDB refusing to start — the error log is a wall of “Database page corruption on disk or a failed file read of tablespace” messages, cycling through table after table. A MariaDB Galera cluster lost a node during a rolling upgrade, the node came back with mismatched GTIDs, and now the ibdata1 on the surviving primary is throwing “assertion failure in thread X in file page0cur.cc.” A small business bookkeeping application built on MySQL 5.7 has been running on a single mirrored pair of consumer SSDs for six years, and today one SSD is unreadable and the other is throwing SMART reallocation warnings.

We recover MySQL and MariaDB databases when the storage under them fails. Both platforms share the InnoDB storage engine and the MyISAM legacy engine, and the recovery approach is the same for both: image the underlying storage, reconstruct the filesystem, extract the .ibd tablespaces or MYI/MYD file pairs, then walk them at the page or record level to pull out what the DBMS refuses to serve.

What breaks in an InnoDB page when storage fails

InnoDB stores data in fixed-size pages, 16 KB by default. Each page begins with a 38-byte FIL header carrying the page number, the tablespace ID, the log sequence number, and both the old-style and new-style checksums. Below that sits the page header with N_HEAP records, B-tree level, and index ID. Below that sit the infimum and supremum sentinel records, then the actual user records in ascending clustered-index order, chained through NEXT pointers. Free space grows toward the middle. The page directory sits above the free space, and an 8-byte FIL trailer at the very end of the page carries a second checksum that must match the FIL header.

When storage returns a page that has been torn by a partial write, one of two things happens. Either the FIL header and FIL trailer disagree — the classic checksum mismatch that produces “Database page corruption on disk” — or both checksums pass because the tear happened on a boundary that preserved them, and InnoDB tries to walk the record chain and hits an assertion failure inside page0cur.cc, buf0buf.cc, or one of the row0 modules. When RAID drops mid-write, entire pages can arrive containing bytes from two different logical pages. When an SSD’s flash translation layer gets confused, InnoDB will read a page that checksums valid but contains stale data from a wear-leveled block that was reassigned to a different logical location.

Error signatures we see repeatedly

• InnoDB: Database page corruption on disk or a failed file read of tablespace — the canonical checksum failure. Reported at InnoDB startup or during a query against the affected page. When it cycles across multiple pages in the same tablespace, the storage under the .ibd is the source.
• InnoDB: Assertion failure in thread X in file page0cur.cc line … — the record cursor found an inconsistency the code cannot recover from. Common variants reference buf0buf.cc, row0upd.cc, or lock0lock.cc. Almost always accompanies a torn page or a page whose contents belong to a different logical page.
• InnoDB: Cannot find record in index — the clustered index and a secondary index disagree about a row’s existence. Torn writes between the two indexes are the usual cause.
• InnoDB: Log file ./ib_logfile0 is of different size at startup — the redo log was truncated or replaced without a clean shutdown. Frequently caused by well-meaning administrators who deleted the log files after a crash, then discovered InnoDB will not start without them.
• InnoDB: page [page id: space=N, page number=X] log sequence number Y is in the future! — the page’s LSN is ahead of the redo log’s recovery LSN, which means either the redo was truncated or the page was written from a future point in time that no longer exists on disk.
• Table ‘./db/table’ is marked as crashed and should be repaired (MyISAM) — the MYI index file’s crash flag is set. MyISAM has no crash-safe design and any interrupted write leaves this state. REPAIR TABLE often works, but when the underlying storage is bad, it rebuilds indexes over damaged data.

How we recover a MySQL database

Every drive that touched the database is imaged first, at the sector level. Nothing else happens until the imaging is complete and verified. When the storage is a RAID array — and a large fraction of MySQL installations sit on hardware RAID or Linux mdadm — we reconstruct stripe geometry from the imaged data itself rather than trusting the failed controller’s reported layout. The output is a virtual disk that presents the datadir the way MySQL should have seen it.

From the datadir, we extract every .ibd (per-table InnoDB tablespace under innodb_file_per_table), the shared ibdata1 (system tablespace and undo log), the ib_logfile0/ib_logfile1 redo logs, and any MYD/MYI/frm files from MyISAM tables. We then walk the InnoDB tablespaces page by page. For each 16 KB page: verify both checksums, walk the record heap along NEXT pointers, verify the page directory slot count against actual records, and either extract the records intact or flag the page for detailed recovery.

Pages that fail checksum but whose records are still parseable get recovered record-by-record. Pages that are completely destroyed leave a gap in the clustered index, which we document and report. When a table’s .frm dictionary file is missing (rare, but happens after directory corruption), we reconstruct column definitions by analyzing the record patterns across many surviving pages. The deliverable is either a set of .ibd files that mount cleanly into a fresh MySQL instance, or a set of INSERT statements ready to load into a new database.

MariaDB, Percona, and forks

MariaDB, Percona Server, and the various MySQL forks share InnoDB internals with upstream MySQL and share the same on-disk format at the page level. Recovery techniques are identical. Where MariaDB Galera clusters or Percona XtraDB Cluster are involved, we work with whichever node’s storage is most recoverable and rebuild the state from there; SST or IST resynchronization becomes a step the customer’s DBA performs after we return a working node.

What we don’t do

We do not undo DROP TABLE or TRUNCATE when the storage recorded the DDL cleanly. If InnoDB flushed the operation to the redo log and the storage confirmed it, there is no on-disk residue for us to recover from. That is what point-in-time recovery from binary logs is for — we cannot substitute for a functional backup rotation. Where we do help on deletions is when the destructive operation happened during a storage failure — a torn redo log flush, a power event during the drop, or a filesystem crash that left the .ibd and the redo in inconsistent states.

Related recovery services

MySQL and MariaDB databases usually share a host with other services that fail during the same storage event. See our database data recovery hub, Linux data recovery when the host runs Linux, RAID data recovery for the underlying array, and PostgreSQL recovery when both databases co-exist on the same failed host.

]]>