Skip to content

0035. Unreadable documents are counted by cron and reported on the status report

Status: Accepted

Date: 2026-10-30

Context

A stored document can stop rendering in full without anyone touching it, and a site learns about it only from the log (work item #3627802). Each step of the render path decides it for one field item, in this order. BlockNoteDefaultFormatter::viewElements() renders nothing when the item's format is not a BlockNote format, and logs a warning; core's ProcessedText::preRenderText() renders nothing for a format that is missing or disabled, and logs an alert. BlockNoteFilter::process() renders nothing when DocumentParser::parse() refuses the value, which covers a version this release does not read (a downgrade after a later release migrated the documents), a value that is not a well-formed envelope and a document over the byte, block or depth limit, and when DocumentValidator::findTableLimitViolation() finds a table over a table limit, and logs a warning. DocumentRenderer::getPlugin() then leaves out a block whose type no plugin provides or the format does not enable, with a debug message a default site does not record. docs/extending.md section 5 said that the administrator should be told what disabling a block costs, and nothing told them. The stored value is never rewritten, so none of this is data loss; it is pages rendering without some of their content.

DocumentValidator::findUnsupportedBlockTypes() already answers the renderer's block question for a whole document, with the two buckets the save path's constraint uses: types no plugin provides and types the format does not enable. Inline content has a milder form of the same problem, a type that renders as its text without its element, and no such finder exists for it: validateContent() reports a type the format does not enable among every other policy problem, and passes over a type no plugin claims.

The renderer also leaves out content inside a block that it cannot read, without a message: BlockNoteBlockBase::buildContent() builds nothing for a block of inline content whose content is not an array, such as a paragraph whose content is a string, and InlineContentRenderer::build() skips an item that is not an array, an item with no type, the editor's carrier type and a text item whose text is not a string. DocumentValidator::validateContent() is the only code that checks content shape, and it answers what a save may store, not what the renderer drops: it checks each block and inline item against its plugin's contract and the format's policy as well, stops at ten problems, and refuses shapes the renderer still renders, such as plain content stored as a string, a link nested in a link, and a link whose target is refused, whose text the renderer keeps.

Counting such documents means reading every revision of every blocknote_document field of every entity type, because a pending revision is what content moderation shows and publishes, and a past one is what a revision page shows and a revert restores. Nothing bounds how many there are. Only some of them are what a site shows or is about to publish: an entity's default revision, and for each translation the revision core's EntityRepository::getActive() loads, which is the latest revision that changed the translation unless that one was saved as a default revision and no longer is, and the latest revision of the entity otherwise. An editor who corrects a document saves a new revision and leaves the old one as it was, so a count of every revision can only fall when revisions are deleted. hook_runtime_requirements() runs each time the status report is viewed, in drush core:requirements, and on /admin/config for an account with administer site configuration, where SystemController::overview() looks for an Error, so the reading cannot happen there.

Canvas has the precedent the work item names, read on its 1.x branch at commit 03ca25a3. _canvas_data_health_requirement() in canvas.install reports how many items cron has checked, how many are invalid and as of when, and points to a canvas:doctor Drush command for the rest. Its cron hook, HealthHooks::cron(), keeps a cursor in state and checks a window of entity_update_batch_size items, 50 by default, per cron run, loading each entity, and keeps one result per item in a table of its own.

Core 11.4 writes every revision ContentEntityStorageBase::loadMultipleRevisions() reads from the database to the persistent entity cache (setPersistentRevisionCache()), so a pass that loads every revision puts every revision of every host into cache_entity, and runs every load hook for each.

Shared storage copies a field that is not translatable into the row of every translation, and a bundle can turn translation off for a base field whose storage has it on, through a BaseFieldOverride, so the bundle's own field definition decides which rows hold the same value; dedicated storage writes such a field in the default translation's row only (SqlContentEntityStorage::saveToDedicatedTables()). Core's State is a cache collector: State::set() writes the key/value row and then rewrites the state entry in cache.bootstrap, which every request reads. Core's lock service is released at the end of the request that took it, and expires after its timeout otherwise.

Decision

The maintainer chose the first of the three options the work item lists: a stored count, refreshed in batches by cron and by a Drush command, and read by a runtime requirement.

  • What is counted. DocumentScanner asks of each stored value, a field item in one revision and, where the field is translatable, in one translation, what the render path asks, in its order, with the module's own code: whether its format is one of FormatRegistry::getFormatIds(), the enabled BlockNote formats; what DocumentParser::parse() answers; whether DocumentValidator::findTableLimitViolation() finds a table; and what DocumentValidator::findUnsupportedBlockTypes() finds against FormatRegistry::getEnabledBlockIds(). The answer is one of the six cases of UnreadableReason: Format, Version (DocumentError::UnsupportedVersion), Limit (TooLarge, TooManyBlocks, TooDeep and a table limit), Malformed (every other parse failure), and MissingBlock and DisabledBlock, which can both apply to one document. The scanner never decides any of these itself. Inline content is not counted, and neither is content inside a block the renderer cannot read: the scanner checks the format, the envelope and block structure the parser checks, the size and table limits, and the block types, and says so on the report.
  • Current and older revisions. A value is current when its revision is the entity's default revision or the revision EntityRepository::getActive() would load for its translation, found the same way from the revision tables: the latest revision whose revision_translation_affected is set for the translation, kept when its revision_default is not set or it is still the default revision, and the entity's latest revision otherwise. For an entity type that is not translatable it is the latest revision. Values in current revisions are what the warning, the value and the per-reason lines count. Values in every other revision are counted apart, as one number, and do not raise the warning.
  • How it is read. The values are read from the field's own table, as the storage's table mapping names it: the dedicated revision table, or the dedicated data table for a field that keeps no revisions, or for a base field in shared storage the revision data table, the revision table, the data table or the base table. Only the value and format columns are read, with the keys that name the row, and the value only when OCTET_LENGTH() says it is no larger than the parser's max_bytes: a larger one is counted as Limit from its length and never fetched. A row whose value is NULL is an empty field, and a dedicated row with deleted set waits for purging. In shared storage, a field that is not translatable on an entity's bundle is read from the default translation's row only, with a join to the base table for the bundle when the field is translatable on some bundles and not on others. The default revision of each entity read comes from the base table and the rest of what decides a current revision from the revision and revision data tables, a few queries per batch. An entity type whose storage is not SQL is not read, and the report names its fields.
  • The cursor. The fields are read in the order of their entity_type.field_name names, and each field in the order of its revision id, or its entity id when it keeps no revisions. A batch reads up to entity_update_batch_size revisions, 50 unless settings.php sets it, with every translation and delta of each, and stops before a revision that would take it past 1,000 values or 8 MiB of fetched values (DocumentScanner::BATCH_ROWS and BATCH_BYTES); it always reads one revision, however large. The pass, with the field and the last id read, is stored after each revision, so a request that ends in the middle of a batch, out of memory or out of time, loses at most the revision it was reading. A field deleted since is skipped, and the pass goes on from the field after it. No setting is added under 0016: the batch size is core's own.
  • Cron. hook_cron() continues a running pass. With none running it starts one when no pass has finished yet, when the last one started a day ago or more, or when the last one's fingerprint no longer matches. It reads until the pass finishes or five seconds have passed, checking the time before each revision, and always reads one revision. It reads nothing while another process holds the lock.
  • The fingerprint. A pass keeps a hash of everything the answers depend on besides the value: DocumentScanner::VERSION, which a change to what the scanner counts raises, the schema version, the document limits, the block plugin ids, and each enabled BlockNote format with the blocks it enables. A batch that finds the running pass's fingerprint changed starts the pass over, and a finished pass whose fingerprint changed is reported as out of date.
  • Storage. The key/value collection blocknote.document_scan holds the running pass as current and the last finished one as last. It is not state, because the running pass is written after every revision read and each State::set() rewrites the state cache every request reads. Each pass holds when it started and finished, its fingerprint, its cursor, its counts per field (checked, unreadable and each reason for current revisions, checked and unreadable for older ones), up to ten block types per block reason and whether it met more, and the fields it could not read. A finished pass is written as last after current is deleted, so a request that ends between the two cannot leave a finished pass to be read again. Uninstalling the module deletes the collection, in hook_module_preuninstall() for the module itself, because core calls hook_uninstall() only as a function in a .install file and the module has none.
  • The lock. Cron and the Drush command take the lock blocknote.document_scan for as long as they read, for 60 seconds at a time, renewed after each batch. Cron skips its run when another process holds it, and checks again under the lock whether a pass is still due. The Drush command waits up to 30 seconds for it and otherwise fails with a message. checkBatch() takes no lock; its caller holds it.
  • The requirement. BlockNoteDocumentScanHooks::runtimeRequirements() reports the last finished pass, or the running one when none has finished or the running one has already found something in a current revision. Its title is BlockNote documents. The value says how many of the documents in current revisions do not render in full, or that all of them do, or Not checked yet. The description has one line per reason found in current revisions, naming how many documents and in which fields, each written to 0023's shape, what is wrong and then what to do: a block a plugin provides is named by its label and one no plugin provides by its type, the format's policy is "does not permit", a limit is what "this site stores", and a document that is not well formed was written outside the editor. Where the fix is to edit the document, the line says to save it, which makes a new current revision. It then says when the pass finished or started; that each translation counts as a document of its own, in its current revision and in the draft waiting to replace it; which checks run and that the content inside each block is not checked; how many documents in older revisions do not render in full, that those are not counted above and that the count clears only when those revisions are deleted; when what the pass depends on has changed since, whether cron is checking again or will on its next run; and, when something was found, that drush blocknote:scan-documents lists them. Fields are named entity_type.field_name, as their storage configuration is.
  • Severity. Warning when a document in a current revision does not render in full, OK when a finished pass found none there, and Info when nothing has been checked yet or a running pass has found nothing so far. Documents in older revisions never raise it. Never Error.
  • The Drush command. drush blocknote:scan-documents starts a pass, runs it to the end and lists each document that does not render in full, in every revision, by field, entity, revision, whether the revision is current, language and delta, with its reasons and block types. A pass that starts over while it runs, because what it depends on changed, empties the list and lists again from the start. Its pass is the one the report reads.

Alternatives considered

Reading default revisions only. Rejected. It misses a document that exists only in a pending revision, which is the one an editor is about to publish, and the count is still unbounded, by entities instead of revisions. Every revision is read, and the warning is based on the default revision and the revision each translation would publish, which takes in the pending one.

Basing the warning on every revision. Rejected. An editor who follows the line's advice, removing a block or saving in a BlockNote format, adds a revision that renders in full and leaves the old one as it was, so the warning would clear only once the old revisions were deleted. The older revisions are still counted, apart, so that a revert or a revision page that would show such a document is not hidden.

A Drush command alone, with no status report entry. Rejected as the only route, because an administrator who never runs Drush is never told, which is the problem the work item states. It is kept as the way to check everything at once and to see which documents they are.

Reading on the status report. Rejected. The reading is unbounded, and the requirements run on every view of the report and on /admin/config.

One result per document in a table of its own, as Canvas keeps. Rejected. It would keep the report current item by item and could name each document, but it needs a schema, and so a .install file the module does not have, plus a row for every revision of every document. A count is what the maintainer chose, and the Drush command names the documents.

Loading revisions through the entity API, as Canvas does. Rejected. It would read any storage and reuse the field item's own parse, but each revision loaded is written to the persistent entity cache and runs every load hook, so a pass would fill cache_entity with every revision of the site. Reading the table costs the module storage that is not SQL, which the report names.

Starting over from hooks. Considered first: hook_filter_format_update(), hook_modules_installed() and hook_modules_uninstalled(). Rejected for the fingerprint, because no hook fires for a code deploy that adds or removes a block plugin, for a downgrade of this module, or for a changed blocknote.limits parameter, and the fingerprint is computed from what is in effect.

One batch per cron run, as Canvas and core's field purge do. Rejected. At 50 revisions a run, a site with 100,000 revisions needs 2,000 cron runs for one pass, close to three months at one run an hour, and the count it reports would be out of date before it finished.

Error severity. Rejected. Error puts "One or more problems were detected with your Drupal installation" on /admin/config for every administrator, and lists the entry among the errors drush core:requirements --severity=2 shows. The site works and the stored documents are intact. Canvas reports invalid data as a Warning too.

Counting inline content rendered without its element. Deferred. The author's text still renders, and there is no finder to reuse: counting it means adding one beside findUnsupportedBlockTypes() that walks inline content the way the renderer does, in block content, table cells and links.

Counting content the renderer cannot read with validateContent(). Rejected. It is the save path's question, so it would count documents that render in full, a link nested in a link or plain content stored as a string, and it stops at ten problems and runs every plugin's own validation. Counting what the renderer leaves out needs a finder that walks block and inline content the way BlockNoteBlockBase::buildContent() and InlineContentRenderer::build() do, the same work the deferred inline finder above needs. Until one exists, the report says that the content inside a block is not checked.

Keeping the passes in state. Rejected. The pass is stored after every revision, and each State::set() rewrites the state cache entry in cache.bootstrap, so a cron run would rewrite it hundreds of times while every other request reads it.

No lock. Rejected. Cron and the command advance the same cursor, so two of them reading at once would each skip what the other read, and a command's listing would fall short of the count it reports.

Consequences

A cron run spends five seconds on the pass plus the time one revision takes, since it checks the time before each revision and reads the one it has started, and between passes nothing but computing the fingerprint and comparing it, without taking the lock. A batch makes two queries per field it reads, one for the next ids with the count and size of their values and one for their rows, and a few more for the current revisions of the entities it read, and parses each value once. Its memory is bounded by 1,000 rows and 8 MiB of values, no value above max_bytes is ever fetched, and the one exception is a single revision holding more than that, through translations and deltas, which is read whole. Each revision read costs one key/value write.

The report lags the site. It says when it was checked; a change to a format, the block plugins, the limits, the schema version or what the scanner counts starts the count over, and a value written without validation, by a migration or a plain save(), is seen by the next daily pass. A running pass that has already found something is shown before it finishes, so disabling a block shows on the report from the first cron run that reads a document holding it.

The count is of stored values, not of documents: a translation whose default revision and pending draft both hold a block no plugin provides counts twice, and the report says that a translation is counted in both. A document that once failed and was corrected stays in the older count until its old revisions are deleted.

A field whose entity type does not keep it in SQL tables is not checked, and the report names it. A field deleted since a pass read it is dropped from the report at once, and its values waiting for purging are never read.

A Drush run holds the lock for the whole pass, so cron reads nothing while it runs. A request killed before its shutdown functions run, which is when core releases a lock, leaves the lock held for at most 60 seconds.

Inline content types that render as their text without their element are not reported, and neither is content inside a block the renderer cannot read, such as a paragraph whose content is a string, which only a value written without validation can hold.

A check that throws is counted rather than retried. Amended 2026-10-31. The cursor is stored after each revision read, so a revision whose check threw, a plugin that fails while being looked up for one block type, left the cursor in front of it, and every cron run read it first, threw again and ended, so the pass never got past it and the report never finished. DocumentScanner now catches any \Throwable around the check of one value, an \Error such as a \TypeError as much as an exception, because either stops the pass the same way and a document that is not as expected is the likelier cause of a type error than a defect that should stop everything. The value is counted as unreadable for a seventh reason, UnreadableReason::Failed, in the same counts as the other reasons, logged at error level on the blocknote channel with Error::logException(), which keeps the class, message and backtrace, and reported to the Drush command's callback with the class and the first 300 characters of the message under error, and the cursor moves past its revision as it does for any other. A defect in a plugin fails every document that reaches it, so only the first failure of each kind in a pass is logged, a kind being the exception class and the line that threw. The pass counts the others under errors, and the Drush command's closing warning says how many repeat a logged one. A pass that starts over logs each kind again. Nothing is reset after the catch: the check opens no transaction and saves no entity, and what it does write, the config entities loaded for the formats and the plugin managers' discovery caches, is correct data that a throw leaves as it was. What could fail in it is the parser, the validator and the plugin managers. A failure that affects every document, such as a broken plugin definition, throws in fingerprint() before the first value is read and is not caught, so the pass does not mark every document failed. The reason is its own, and not Malformed or one of the others, because the scanner does not know whether the document renders: the line on the status report says it could not be checked and so may not render in full, points to the log, and asks for a report to the issue queue or the module the error names. It counts toward the warning and the total, which then say that the documents do not render in full, because a document the checker cannot read is more likely than not one the renderer cannot either, and a quiet OK would hide it. The Drush command lists it with the reason failed and an Error column, and ends with a warning saying how many could not be checked. Fatal errors that PHP cannot catch, such as running out of memory, still end the request and still leave the cursor in front of the revision. A DocumentScanner takes the blocknote logger channel as its last argument.

The requirement's strings are new strings to translate, plural forms included. README.md describes the entry under "The status report", and docs/extending.md section 5 now says the status report counts the documents a cleared checkbox affects.

References

  • This module: src/Document/DocumentScanner.php, src/Document/UnreadableReason.php, src/Hook/BlockNoteDocumentScanHooks.php, src/Drush/Commands/BlockNoteCommands.php, src/Document/DocumentParser.php, src/Document/DocumentValidator.php (findUnsupportedBlockTypes(), findTableLimitViolation(), validateContent(), label()), src/Plugin/Filter/BlockNoteFilter.php (process()), src/Plugin/Field/FieldFormatter/BlockNoteDefaultFormatter.php, src/Render/DocumentRenderer.php, src/BlockNoteBlockBase.php (buildContent()), src/Render/InlineContentRenderer.php (build()).
  • Tests: tests/src/Kernel/DocumentScanTest.php, tests/src/Kernel/ModuleUninstallTest.php.
  • Work item: https://git.drupalcode.org/project/blocknote/-/work_items/3627802
  • Canvas: https://git.drupalcode.org/project/canvas/-/blob/03ca25a3f80230938f7bb96ba2953715aec30a5f/canvas.install, src/Hook/HealthHooks.php and src/Health/Doctor.php (runCronWindow()) at the same commit.
  • Core 11.4: ContentEntityStorageBase::loadMultipleRevisions(), setPersistentRevisionCache() and getLatestTranslationAffectedRevisionId(), EntityRepository::getActive() and getLatestTranslationAffectedRevision(), SqlContentEntityStorage::countFieldData() and saveToDedicatedTables(), State::set(), DatabaseLockBackend, SystemController::overview(), ModuleInstaller::uninstall(), ProcessedText::preRenderText().
  • 0001, 0002, 0016, 0023.