0039. Stored text converts in place into a document field beside it¶
Status: Accepted
Date: 2026-10-31
Context¶
0038 converts one value into one document and stores nothing. A site moving a body field from CKEditor to BlockNote has every value of the field to move, in every translation of every revision: the early user has 212,047 bodies, Georgian only, with about 950 older revisions kept between them, has not decided whether to convert those, and wants a pass after its migration that can be run again after every reset of its database, with a report per entity and per element to compare between rehearsals.
A text field cannot become a document field in place: the field types differ, and so do their columns. A revert restores every field of the revision it reverts to, so a revision whose document field is empty shows an empty body once reverted, and the old field, once deleted, takes every value of every revision with it.
Core 11.4.11 saves a revision where it is when the revision is saved with setNewRevision(FALSE): ContentEntityStorageBase::doPreSave() loads the revision being saved as the original for a non-default revision. setSyncing(TRUE) keeps ChangedItem from setting the changed time, and keeps content moderation's ModerationHandler::onPresave() and ModerationStateFieldItemList from making a new revision or changing the default one. Content moderation still sets the publishing status from the moderation state, and reads the state of a revision saved before the bundle was moderated as the workflow's initial one, which ContentModeration::getInitialState() gives as published for a published revision, so a revision keeps its status; EntityOperations::entityUpdate() then records a state for the revision, which it had none of. ContentEntityStorageBase::populateAffectedRevisionTranslations() recomputes the affected flag of every translation whose stored flag is NULL, which is how core stores "not affected", and marks each one affected whose fields changed, so writing a document into every translation would mark every one affected and move getLatestTranslationAffectedRevisionId().
A document's identity, the field item's uuid, is minted on first save and kept; BlockNoteDocumentItem::preSave() mints another when the identity is stored on another entity, and its lookup leaves out the host's own revisions. blocknote.document_saved fires for each document a save changed, after the commit, with the change origin the writer marked (0009). Core's database lock is released at the end of the request that took it, and expires after its timeout otherwise. The persistent entity cache in 11.4 keeps every revision loadMultipleRevisions() reads, under a tag resetCache() invalidates.
The early user's notes ask for a Drush pass rather than a migration, and have no summaries.
Decision¶
A new field beside the old one, filled in place. drush blocknote:convert <entity_type> <source_field> <target_field> converts the values of a text, text_long or text_with_summary field into a blocknote_document field of the same entity, in the BlockNote format --format names, or the one format the target field allows. Each revision is written where it is: the revision is read again under its host's row lock, inside a transaction, and when nothing it was converted from or into has changed since it was read, the target field alone is set and the revision is saved with setNewRevision(FALSE) and setSyncing(TRUE), after setRevisionTranslationAffected(FALSE) on each translation whose flag is not set. No revision is added, no default revision moves, no changed time changes, no translation becomes affected, and the source field is never changed; deleting it is the site builder's own last step. The conversion itself runs before the transaction, on a copy of the revision, which a rule may read.
Current revisions first, older ones as a second pass. --revisions=current, the default, converts each entity's current revisions, as 0035 defines them: the default revision, and each translation's active revision, the latest revision that changed it unless that was saved as the default revision and no longer is, and the latest revision otherwise, which is where content moderation's draft is. Every translation of such a revision is converted. A current revision is written only when the entity's latest revision is still the one it was when the revision was found to be current. --revisions=older converts every other revision, in the order of its id, as long as the revision is still there; --revisions=all runs both. The current pass is meant for a maintenance window, which is also when the form and the display are switched to the document field, so that no author edits the text field once its value is converted; the older one may run while the site is open, at a quiet time, since each revision it writes invalidates its entity's cache tags and its type's list tag.
Older revisions are written silently. ChangeOrigin::Converter marks every save, and blocknote.document_saved fires for each document written into a current revision. The save of an older revision announces nothing, through ChangeOriginTracker::silence(), which is internal to the converter: a mirror of a document would read an event for an old revision as the document changing back. Nothing fires for a document left as it was.
Identities are derived from where a document is stored. A document's uuid is a name-based UUID, under 0038's namespace, of the entity type, the entity's uuid, the language, the target field and the delta of the value converted, the language being the source language's for a field the translations share; its block ids already are. So a document keeps its identity in every revision and after every reset of the database, and converting a value again writes the same bytes.
A target holding a document is left alone. An empty target is written; one holding the bytes the conversion would write is left as it is, unchanged; any other is skipped_target_differs. --overwrite replaces a document only when the converter wrote it and nobody has changed it since: a ledger in the key/value collection blocknote.conversion_ledger keeps the hash of every document the converter wrote, with the job that wrote it and the hashes of the source values it was converted from, at most twenty, a current revision's kept longer than an older one's. A document the converter replaces is forgotten once no revision of its entity holds it. A field is written whole or not at all: when one of its values in a translation is refused, none of them is written, so that once the value converts the field is written in one piece and no document moves. A value that is empty, or converts into no block, has no document, so the documents after it in a field of several values move up a position; each keeps the identity of the position it was converted from.
A job keeps its place. A job is known by its entity type, source field and target field, node.body.field_body, and a dry run by the same with .dry-run. Its cursor, the last entity id the current pass read and the last revision id the older pass read, is stored in the key/value collection blocknote.conversion after each entity or revision, with the job's totals and the length of each report file, once its rows are written, so a run stopped by --limit, by memory or by a defect is resumed by running the job again, which cuts the report files back to that length and so reports each value once, and --restart starts it over. An entity or revision whose conversion or write throws is reported as failed and passed over; ten entities in a row, the revisions of one entity counting as one, stop the run before the first of them, which the next run reads again, and --max-failures sets how many, 0 for none. A job remembers the format, the bundles, the summary field and the options it was started with, all but the guard's limits, and a run with others is refused. --ids names entities and keeps no place. A run takes the lock blocknote.conversion:<job> and renews it before it converts each entity or revision and again before it records it; another run of the same job stops with exit code 4, and so does a run whose lock expired and was taken. Batches are entity_update_batch_size entities or revisions, 50 by default, and the entity memory cache is emptied after each.
Everything is checked before anything is read. The entity type, the bundles, both fields' types, the formats against the target field's allowed formats, the translatability of both fields, which must match, and the target's cardinality, which must hold as many values as the source's; a check that fails stops the command with exit code 2.
The report is two files and a table. documents.csv has a row for each value, with its outcome, the size of its document, the hash of its source and a count for each category; elements.csv has a row for each report entry of a value in a current revision, and only the refusals of one in an older revision, whose entries would repeat the current revision's. The files go to --report-dir, by default a directory in temporary:// itself named after the job, the time in UTC and sixteen random hexadecimal characters, so that nobody can make it first, and a resumed run appends to the files its job started; a directory where the web server serves files, under the web root or in the directory of public://, assets:// or another stream wrapper of their kind, is refused wherever its path leads, in any case of its letters, since the reports quote content; so is any stream but a local path and a local stream wrapper Drupal registers. The directories made and the files written are their owner's alone, and a directory or report file another user owns or others can write is refused. A text cell a spreadsheet would read as a formula is written after an apostrophe. The console prints the run's outcomes beside every run's, and each code and key with its events and the number of documents it appears in, a code every value of a source format carries counted once per key. The command exits with 0, 1 when a value was refused, 2 for a usage error or a failed check, 3 for a defect, an entity or revision reported as failed included, and 4 for a lock held or taken.
A value cannot hold a run up. Each conversion runs under 0038's converter with a guard: markup larger than --max-source-bytes, four times the site's largest document by default, is refused before it is read; a conversion taking longer than --max-seconds, 10 by default, is refused; and markup the parser would take longer than the time left to read is refused before each time it is parsed, as stored, before each filter of the source format that parses it and before a rule parses markup of its own, through ConversionContext::checkParse(), by an estimate of the parser's work, since masterminds/html5 takes time in the square of how deep elements nest, block elements most, and cannot be stopped once it starts: the markup is read with the parser's own tokenizer, so that every tag, comment and text the estimate reads is one the parser reads, the open elements are kept as its tree builder keeps them, and its walks up them and the depth at which it appends each node are counted; an element carrying more than 256 attributes, which the DOM sets in time in the square of their number, is refused there too, as markup_too_many_attributes. The estimate counts those costs and no others, and takes time in the length of the markup alone. A cost it does not count, of the parser or of a filter of the source format, is refused once the stage that spent it ends, by the time limit, as conversion_too_slow: masterminds/html5 keeps the namespace it pushed for each svg or math element closed without its own end tag, and moves that growing stack each time it opens another, and Xss::filter() reads to the end of the line for each comment that is never closed, both drafted as upstream reports in docs/drafts/. A run held up by such a value stays safe: once the job's lock expires another run may take it, but the lock is checked again before the value's rows and the place past it are written, and its revision is written only when nothing it was converted from or into has changed since it was read. Either limit is 0 to turn it off. A refused value is reported and its source left alone.
The summary is left out and reported unless a field is named for it. A text_with_summary value with a summary reports summary_dropped; --summary-target names a document field to convert it into, in the format --summary-format names or the one that field allows.
Rules take options from the command. --rule-option=name=value, repeated, reaches every rule through ConversionOptions, on both commands; a name of the module's own options is refused there, since each has an option of its own.
The status report says what is left. After a run that finishes the passes it was asked for, the job counts the revisions holding source text and no document, older revisions not converted yet and those whose values were refused or converted into nothing, and the current revisions holding a document the converter wrote from text that is no longer there, which an author changed after the conversion, read from the entities saved since the job started where the entity type records a changed time. The status report says how many of each, in an Info line, while both fields exist, and names the entities of the second for --overwrite --ids: reverting to one of the first leaves the document field empty, and deleting the text field loses the text of both.
One value converts the same way from any caller. The guard, the carrier of 0040 and every report entry about the source format belong to HtmlConverter::convert(), so a caller converting one value at a time, such as a migration's process plugin, gets what the command gets, a host entity and a cursor aside. No migrate process plugin ships with this wave.
Rehearsal¶
No copy of a real site was available, so on 2026-11-01 the procedure of docs/converting.md section 9 ran end to end on the development site, Drupal 11.4.11 on PHP 8.3 without Xdebug and MariaDB 11.8, against a synthetic corpus, nothing of which is committed. It held 20,000 nodes whose text_long body in basic_html, its allowed tags widened to tables, iframes, spans, classes and a custom element as an older site's were, was drawn from the early user's shapes and proportions: Word and Google Docs pastes, nested spans and , lists, headings, related-article lists of absolute links to other nodes, images by data-entity-uuid and by legacy and derivative addresses, YouTube and Vimeo players, video_embed_field's token, social embeds, junk classes and Georgian text, with 20 adversarial values among them. 1,000 nodes had four older revisions each, 24,000 revisions in all; the current revisions held 234 MB of markup, 11.7 KB a value on average and 7.5 MB at most. 300 public files were named, half of them held by image items. The documents were written in a format permitting media, tables and content links, with --create-media, the image type and core's Remote video type, its thumbnail downloads queued. A real site's rehearsal is still to be run.
| Measured | Result |
|---|---|
| One value's conversion | 2.1 ms at the median of 2,000 values, 3.7 ms at the 90th percentile and 9.2 ms at the 99th; 72 ms for the largest realistic value, 334 KB; 1.45 s for the slowest adversarial one, 3,000 nested div elements |
| Dry run of the current pass | 20,000 values in 67 s, 300 a second |
| Current pass, writing | 100 to 117 values a second; 11,000 entities in 110 s |
| Older pass | 24,025 revisions read, 4,025 converted, in 41 s, 99 a second |
| Batch size, dry run over the same 3,000 entities | 10: 13.4 s and 30 MB at peak; 50: 11.5 s and 37 MB; 200: 11.3 s and 49 MB |
| Batch size, writing 3,000 entities each | 10: 29.7 s and 31 MB; 50: 25.6 s and 36 MB; 200: 29.2 s and 88 MB, with a 3 MB value in its range |
| A whole run's peak memory | 93 MB for the dry run of 20,000 values and 110 MB for the pass over 11,000 entities, against 30 to 37 MB for 3,000 entities at a batch of 10 or 50, so it grew with the run; the largest value, of 3 MB and 1.5 MB, set part of it, and the measurements do not separate the two |
| Closing counts | Under a second for a run once its passes are done, 25 entities an author changed included; 19.6 s, about 1,000 entities a second, when every entity holding a document was saved since the job started |
| Report files | documents.csv 4.2 MB for 24,025 rows, 175 bytes a row; elements.csv 7.1 MB for 15,160 rows over 20,000 current values, 0.76 a value, a third of its bytes class_not_mapped; the console's table of codes 40 rows |
| Ledger | 23,323 rows, 3.5 MB |
| Refused | 5 of 20,000 current values, all adversarial: two markup_too_many_attributes, one source_too_large, of 7.5 MB, and two too_large, documents over the 1 MiB max_bytes |
The counts are as measured. The writing pass was resumed in parts, as a job's place allows, and covered 11,000 of the 20,000 entities; neither this record nor docs/converting.md says what ended it there, and neither says what accounts for the ledger's 23,323 rows against the 24,025 revisions the older pass read.
The guard's defaults refuse no realistic value. The slowest realistic value took 72 ms of the 10 seconds a value may take, and four times max_bytes, 4 MiB, refused only the 7.5 MB value; the documents over 1 MiB were refused by the field's own limit, as a save refuses them.
The batch default stays. A batch of 10 costs about a sixth more time than 50, and one of 200 saves none and holds more in memory, while a run's peak is set in part by its largest value; entity_update_batch_size, 50 by default, is kept.
The report stays as it is. The console's table of codes stays short because keys name a tag, a class, a provider or an entity type rather than a value, and elements.csv, at under one row a value, can be filtered by code; two adversarial documents wrote 2,279 of its rows, which are what an author needs to rebuild them.
cache_entity grows by about twice the content converted until the bin is cleared. The dry run left a row for each of its 20,000 entities, 289 MB. The current pass leaves two rows for each entity it writes: the entity's, which is valid and what pages would cache anyway, and the revision read again under the lock, which the save invalidates and which stays as a dead row until the bin is cleared; after 11,000 entities there were 9,347 entity rows of 243 MB and 10,998 dead revision rows of 274 MB. The older pass loads every entity's default revision to tell its current revisions apart, 19,361 rows of 495 MB here for 4,025 older revisions, and leaves a dead row for each older revision it writes, 4,025 of 44 MB. drush cache:clear bin entity after each pass clears them. Amended 2026-11-01. A site rehearsal found "twice the content" too large: saving a revision deletes its entity's own row, as EntityStorageBase::doPostSave() does on every save, so the current pass leaves no valid row for an entity it writes, only the stale row of the revision it read again, and the older pass none for an entity whose older revision it writes. What a pass leaves is roughly a stale row for each revision written and a row for each entity read and not written, as docs/converting.md section 9 says.
Media. 959 players and tokens named 142 remote video items, one for each video, and 165 image items were created; 141 thumbnail downloads were queued for cron.
What pathauto and Search API do with the saves, read in their source rather than run, since neither is on the development site: pathauto 8.x-1.15 generates the alias of a default revision on every save whatever its syncing flag, and leaves any other revision alone, so the current pass rewrites the alias of each entity whose pattern now gives another than the one stored, unless pathauto's update action is "Do nothing"; Search API 8.x-1.43 marks the entity's item as changed in each index tracking it on every save of either pass, without asking whether the revision is the default one or the save is syncing, and an index set to index items immediately registers them to be indexed once the request ends.
What the rehearsal found. A refused value's report still said would_create in drush blocknote:convert-html, and an older revision named an item only an unpublished item held that it had not named while it was current, so a revision converted again once an author's edit made it older was passed over as skipped_target_differs; both are fixed, as 0042 and docs/converting.md describe. A link at an address of the site that names no content, such as the old site's article address no route answers or a deleted node's, stays a plain link without a word, as section 3 of that document says; the corpus's generator gave two links in five the old site's article address, and the report named none of them.
Alternatives considered¶
A new revision for every converted revision. Rejected: it would double every entity's revisions, make each conversion the latest revision, and move content moderation's draft.
Every revision by default. The design's default, rejected after review: the current pass needs a maintenance window and the older one does not, so they are run apart anyway, and a site that keeps many revisions would hold its window open for revisions nobody sees. The older pass is one option away, and the closing line and the status report say how much of it is left until it has run.
Announcing the older revisions' documents too. Rejected: a subscriber mirroring documents would read each event as the document changing to what an old revision holds, and on a site that keeps many revisions the older pass writes many times more documents than the current one.
Keeping the identity the field item mints. Rejected: each revision's save would mint a different one for the same document, and every reset of the database another.
Writing a field's converted values and leaving out the refused one. Rejected: the documents after it would move up a delta, and their identities and block ids, which are derived from the delta, would no longer line up with their source.
Replacing any document with --overwrite. Rejected: a document an author wrote since the conversion would be lost. The ledger limits it to the converter's own.
A migrate process plugin now. Deferred: the early user asked for a pass after its migration, a migration's destination entity does not exist when a value is processed, so neither the identity nor the block ids could be derived from where the document is stored, and each --update of the migration would give the document a new identity. The per-value API is ready for one when a site asks.
A Batch API form. Deferred, as 0038 planned: a form needs an access and interface design of its own, and 212,000 bodies are a command-line job.
--format-map, a target format for each source format. Deferred until a site converts from more than one format into different ones.
Interrupting the parser with a signal. Rejected: pcntl is not everywhere a conversion runs, and a signal handler interrupting core's parser in the middle of building its tree is a defect of its own. The estimate refuses exactly the shape that makes the parser slow.
Estimating every cost of the parser and of the filters. Rejected after the third review of the guard: each review found another path where masterminds/html5 or core's filters take time in the square of the markup, the estimate would have to follow their internals release by release, and masterminds/html5 2.12.0 changed the walk the estimate counts. The guard bounds the module's own work and refuses every value that overruns; the time limit catches the rest after the fact.
Estimating from the tags with patterns of its own. The first estimate, rejected after review: a reader of tags that is not the parser's ends a comment, a tag or an attribute value elsewhere than the parser does, and markup shaped to make it take deep nesting for a comment or an attribute, or end tags inside a CDATA section for real ones, kept the estimate near nothing while the parser took seconds.
Consequences¶
A site converts a field in four steps, which docs/converting.md describes: add the document field, rehearse with --dry-run and the reports, convert the current revisions and switch the form and the display to the document field in a maintenance window, and convert the older ones, or decide not to, before the text field is deleted.
A revision converted with --revisions=current only keeps an empty document field in its older revisions; the command's closing line and the status report say so until the older pass runs.
Each converted current revision fires one blocknote.document_saved per document, with the origin converter, which docs/workflows.md section 10 adds to the three origins of 0009; the older pass fires nothing, which section 1's "observing the documents a save changed" now excepts. The "three cases" sentence of 0009's change origin paragraph is superseded.
Writing a revision in place stores the affected flag of a translation the revision did not affect as 0 where core stored NULL; core reads both the same. A moderated host's revisions saved before the bundle was moderated each gain a moderation state. Contributed modules that react to a save see each converted revision saved: pathauto may rewrite the alias of a default revision, and Search API marks the entity changed for indexing on every save of either pass, as Rehearsal reads in their source, so a site sets pathauto's update action and its indexes' immediate indexing for the window, and indexes afterwards.
The key/value collections blocknote.conversion and blocknote.conversion_ledger grow with the jobs run and the documents written, a row for each document stored, of at most twenty source hashes; uninstalling the module deletes both. A document dropped from the ledger's sources whose current text was among them is counted as changed after the conversion, and converting it again with --overwrite writes the same bytes and records its text again. Counting the text changed after the conversion reads the entities a page of revisions at a time.
Every revision read is written to the persistent entity cache, as core does for any revision load, and invalidated by the save that writes it or left there for one passed over; a run leaves cache_entity holding about twice the content it converted, a valid row for each entity and a dead one for each revision written, until drush cache:clear bin entity clears it after the pass, as Rehearsal measured. Amended 2026-11-01. Less than that: the save deletes the row of the entity it writes, as Rehearsal now says. Each revision the older pass writes is saved, and the save invalidates its entity's cache tags and its entity type's list tag, such as node_list, so a site whose pages list content builds them again after each revision written: the pass is run at a quiet time.
References¶
src/Convert/Bulk/(ConversionJob,JobPreflight,BulkConverter,DocumentWriter,ConversionLedger,DocumentRow),src/Convert/Report/CsvReportWriter.php,src/Convert/Report/ReportDirectory.php,src/Convert/Report/ReportSummary.php,src/Convert/ConversionGuard.php,src/Convert/ParseEstimate.php,src/Convert/ConversionSource.php(documentUuid()),src/Drush/Commands/BlockNoteConvertCommands.php,src/Hook/BlockNoteConversionHooks.php,src/Document/ChangeOrigin.php,src/Document/ChangeOriginTracker.php(silence()),src/Hook/BlockNoteDocumentEventHooks.php.tests/src/Kernel/Convert/BulkConversionTest.php,BulkConversionModerationTest.php,ConvertCommandTest.php,ConversionGuardTest.php,ReportDirectoryTest.php;tests/src/Unit/Convert/BulkPartsTest.php,ConversionGuardTest.php.- Drupal core 11.4.11:
ContentEntityStorageBase::doPreSave(),populateAffectedRevisionTranslations(),resetCache(),ChangedItem::preSave(), content moderation'sModerationHandler::onPresave(),ModerationStateFieldItemList,EntityOperations::entityUpdate(),ContentModeration::getInitialState(),EntityRepository::getLatestTranslationAffectedRevision(). masterminds/html5 2.10:Tokenizer,DOMTreeBuilder::startTag(),endTag(),autoclose(),TreeBuildingRules. - docs/converting.md, docs/workflows.md sections 1, 10 and 11, docs/extending.md section 14.
- 0009, 0035, 0038, 0040.