0038. Formatted text converts to a document through a PHP walker whose rules are tagged services¶

Status: Accepted

Date: 2026-10-31

Context¶

A site moving a field from CKEditor to BlockNote has formatted text to move with it, and nothing in the module read HTML other than a browser paste, through upstream's own parser in the editor. The maintainer wants a converter before 1.0.0. The backlog item "Convert existing formatted text into documents" set out eleven questions; a design answering them and a second-opinion review of that design were both accepted on 2026-10-31, the review's changes winning where the two differ. The work is split into waves, and this record covers the first two, 1a and 1b, the conversion of one value into one document; writing documents into fields, resolving entities and media, and carrying markup that cannot be converted have records of their own when they land.

Upstream's HTML parser, tryParseHTMLToBlocks(), drops exactly the Drupal attributes a conversion exists for: data-entity-uuid, data-align, data-caption, <drupal-media>, the text-align-* classes and every link attribute but href. It needs a DOM, so it would run under Node, and only PHP has the entity API, the format's policy and the document validator in the same process.

Core's Html::load() parses with masterminds/html5 2.10, a pure PHP HTML5 tokenizer whose tree builder has neither the adoption agency algorithm nor foster parenting. Measured on the repository's own vendor/: xy becomes xy, where a browser builds xy; stray text in a table row stays in the row; namespaced and custom elements such as <o:p> survive; a numeric character reference to a surrogate is decoded to bytes that are not UTF-8, one to 0 ends the text it is in, one to a C1 control is read as that control, and one past U+2FFFF is left as text, where a browser reads U+FFFD for the first two, the Windows-1252 character for the third and the character itself for the fourth; a named reference without its semicolon, such as &copy or &AMP, is left as text, where a browser reads the 106 names the standard keeps from HTML 4 without one, and a few names no browser knows, such as &am;, are decoded; a NULL in text is read as U+FFFD, where a browser drops it, and every other control character is kept, as a browser keeps it and draws no letter for it; and a CDATA section is kept as text, where a browser reads a comment. PHP 8.4's Dom\HTMLDocument follows the specification, and the module's floor is PHP 8.3.

The editor reads formatting from markup with tiptap's parse rules and the module's own marks:  is bold unless it declares font-weight: normal, a font-weight of bold, bolder or 500 and above sets bold and one of 400 clears it, font-style: normal clears italic, text-decoration sets underline and strike, vertical-align sets subscript and superscript (0031). Google Docs wraps everything it copies in . BlockNote reads a block element's color and background-color as the block's own props, and its text-align as the block's alignment.

The editor writes a document back in one canonical shape, and a document in any other shape is rewritten the moment it is opened: every prop a block type declares with a default is written, an optional prop such as a list's start or an image's previewWidth only when set; adjacent runs with the same styles are one run, adjacent links to one target one link; a line break is written onto the end of the item before it, whatever that item's styles, or as an unstyled run of its own when nothing comes before it (nodeToBlock.ts in BlockNote 0.57.0). Since 0.57.0 the editor also refuses to open a run carrying marks its schema excludes, which 0037 made the server refuse too.

BlockNote 0.57.0 holds a table as rows of cells, each with its colspan, rowspan, colors and alignment, and headerRows and headerCols counting the leading rows and the leading positions in a row whose cells are headers; it writes them back by counting the rows whose every cell is a header and the positions where every row's cell is one (nodeToBlock.ts), so the two counts survive a load only for some tables. A cell holds inline content only. When a table is mounted, prosemirror-tables fills a row shorter than the table with empty cells and shortens spans that overlap. A details element is a toggle list item, or a heading whose isToggleable is set where the format permits a collapsible heading.

A page showed stored markup after the filters of its format had run. Core's filter_html runs Xss::filter(), which removes the tags the format does not allow but keeps their text, and escapes an ampersand that starts no reference ending in a semicolon, so that a page under it showed &copy as written, then parses and serializes the result with Html::load(). _filter_autop() is deprecated; filter_autop, filter_url and filter_html_escape have settings and an order of their own in each format. filter_html_image_secure shows an image only when its source is on the site and getimagesize() reads the file on the server rendering the page.

The constraint checks a document in a fixed order: the parser's limits, then findTableLimitViolation(), findUnsupportedBlockTypes(), findSingleParagraphViolation() and validateContent(). An entity's validate() adds the format permission, which core checks against the acting account, and every other field's constraints on legacy data.

BlockNoteBlockBase::propSchemaFor(), the property schema narrowed to a format's settings, such as a heading's permitted levels and a code block's languages, was protected. It keeps a property the editor no longer writes in a format, such as a heading's isToggleable where collapsible headings are off, because a stored document may still carry it.

Decision¶

The converter is PHP, over core's Html::load(). Drupal\blocknote\Convert\HtmlConverter, the service blocknote.html_converter, converts one value for one BlockNote text format and returns the document and its report; it stores nothing. The parser's divergences from a browser are accepted and measured on real content before the converter is finished, because output that depends on the PHP version would be worse than a known divergence.

Rules are tagged services, and they are the one seam. A conversion rule is a service tagged blocknote.conversion_rule with a priority, the higher asked first, and implements one or more of six interfaces in Drupal\blocknote\Convert\Rule: an HTML preprocessor repairs the stored string, a DOM preprocessor changes the parsed tree, a class map provider says what classes mean, a text pattern rule converts a block whose whole text has a shape of its own, an element rule converts the elements of the tags it names, and a document postprocessor changes the finished blocks. Amended 2026-10-31. 0041 adds two more, a file resolver and a link resolver, on the same tag. The module's own rules are registered the same way, at priority 0. ConversionRuleRegistry takes them from a !tagged_iterator, splits them by interface and indexes element rules by tag. There is no plugin type, no alter hook and no event: no configuration names a rule and nothing lists rules to an administrator, so a plugin manager, a cache and derivers would buy nothing, and a second way to change a conversion would be a second contract, the posture 0009 takes toward a pre-save event. Whatever observes a converted document observes it the way it observes any other write.

What an element rule returns decides where it goes. An ElementResult is blocks, inline content, a replacement by nodes or by markup, an unwrap, a drop or a strip. Blocks returned for an element met inside a paragraph split the paragraph around them, and inline content returned for an element met between blocks becomes a paragraph of its own; so an image in a sentence splits it, and a custom element holding an embed code decodes it with replaceWithHtml() and lets the rules for what it decodes to take it. A replacement is walked by every rule again, at most eight deep and at most 1000 times in one document, after which the element is dropped and reported. A rule hands the walker only what is inside the element it converts: asking it to convert that element, or one holding it, would ask the rule again without end, so the walker throws a \LogicException, and an element nested more than 10000 deep is dropped and reported, which stops a rule that keeps making the markup deeper. The walker treats an inline element holding blocks, such as a  a word processor wrapped around paragraphs, as transparent, its formatting carried into the blocks inside it, and the props a class mapping gives it into its own content only: a paragraph collected across its edge is split where the props change. An element no rule converts is unwrapped when it holds content and dropped when it holds none.

The target format's policy is applied while building. Every block and every piece of inline content is built through BlockFactory, which reads the format through TargetPolicy: a block holding text that the format does not enable becomes a paragraph, any other is dropped; a prop value the format refuses becomes the nearest permitted number, the larger of two as near, or the default; a style, a color, an alignment, a link, a link's scheme or its new-tab flag the format refuses is left off, the text kept; and a block is built with exactly the props the editor writes in that format, so a heading carries isToggleable only where the format permits a collapsible heading, as the editor's own heading declares it only there. The property schema comes from the block plugin, so the converter never copies a plugin's values nor names a plugin: getPropSchemaFor(), which this record makes public on BlockNoteBlockInterface and BlockNoteInlineContentInterface, and BlockNoteBlockInterface::getEditorPropSchemaFor(), which this record adds, the properties the editor writes in a format, getPropSchemaFor() unless a block overrides it, as the heading does to leave out isToggleable. Marks the editor cannot hold together are separated by 0037's own rule: code gives way to any other style, color, new-tab flag or link, and subscript to superscript. A single-paragraph format's blocks are joined by 0030's rule. Nesting deeper than the site's limit is flattened. Each narrowing is reported.

Inline content is written the way the editor writes it, line breaks onto the item before them, so that every converted document opens in the editor unchanged.

Marks are read the way the editor's paste reads them. The tag and style rules above, the Google Docs wrapper included, and no others. A color converts only when it is a palette name, in a declaration, a  or a data-text-color or data-background-color attribute, the British spelling of gray read as gray; there is no nearest-color matching, because a guessed color is a silent change. On a block element a color and an alignment are the block's props, from text-align or align; a container passes its alignment and colors to the blocks inside it, as CSS inheritance shows them. Whitespace collapses as a browser shows it: a single no-break space between two visible characters stays one, a run of spaces and no-break spaces becomes one space, whitespace at the edges of a block and beside a line break goes, and a   that ends a block, which draws nothing, is dropped. Preformatted text is kept exactly.

Block ids are name-based, given after the postprocessors. Each id is a version 5 UUID of a fixed namespace, the source's seed and the block's position in document order. The seed is the entity type, uuid, language, target field and delta when the source names an entity, and a hash of the markup otherwise, so converting the same value again writes the same bytes.

The checks a save runs gate the result. ConversionGate encodes the document, parses it under the site's limits, stamps and normalizes it as the document updater does, encodes it again, and runs the validator's checks in the constraint's order on the final bytes. It does not call $entity->validate(), which would check the format permission against whoever runs the conversion and every other field's constraints on legacy data. A refused document must not be stored; a refusal is a site limit, which a large source can reach, or a defect in the converter or a rule. A Kernel test runs the field's constraint over every converted fixture, so the gate and the constraint cannot drift apart.

Everything not carried as found is reported. A report entry has a category, a code, a key, the path where it was met, a snippet of the markup, a detail and the rule that caused it. dropped, degraded, unresolved and refused are listed once per event, except that one narrowing of one block's text is one entry counting each run it concerns; ignored and stripped are counted once per code and key, because they name what is found many times over and changes nothing a reader sees. A class no mapping names is reported, so a site finds the classes its markup relies on, except on an element a rule of another module converted. Amended 2026-10-31. The media submodule's rules count as the module's own here, and 0042 adds a category, created, listed once per event, for an entity the document names that its write creates. A font family is reported by value, because a font that encodes a script shows different letters from the ones stored. data-*, lang, dir, id and the other attributes and declarations the document cannot hold are not reported.

A document is encoded the way the editor writes it, with JSON_UNESCAPED_UNICODE, JSON_UNESCAPED_SLASHES and JSON_UNESCAPED_LINE_TERMINATORS, so that a Georgian letter takes three bytes rather than six against the size limit. A value that is not valid UTF-8 has each bad sequence replaced with U+FFFD and is reported, rather than stopping a run over thousands of values; so does markup a rule returns, and text a rule writes is replaced the same way when the document is encoded. Before any markup is parsed, the stored markup or a rule's, each numeric character reference is rewritten as the one a browser reads, or as the character itself past U+2FFFF; each named reference a browser reads without its semicolon becomes a numeric one, by the standard's rule for text and its stricter one for an attribute, and an ampersand a browser reads as itself is escaped; and a control character, written as itself or as a reference, is removed, or replaced with U+FFFD for a NULL in an attribute, and counted, since a browser draws no letter for it. After the markup is parsed every CDATA section is removed.

drush blocknote:convert-html converts one snippet from a file, standard input or --html, prints the document and the report, and exits with 1 when the gate refuses, 2 when it is called wrongly and 3 when anything throws once its options are read, so that a script tells a refusal from a usage error and both from a defect. Whether --format names a BlockNote format is asked of the format registry before anything is converted, so that an \InvalidArgumentException a block plugin throws while the format's policy is built is a defect too. It is the tool for writing rules.

The source format's filters run before parsing, through its own plugins. When the conversion names the format the markup was stored under, its enabled filter_html_escape, filter_autop, filter_url and filter_html run on the markup after the HTML preprocessors, in the format's weight order and with its settings, after the newline normalization core applies first. filter_html runs as it is, Xss::filter() and its parse included, rather than being mirrored from getHtmlRestrictions(), so that what the format never let a page show is not converted either, even where that parse diverges from a browser. filter_html_image_secure is the one filter handled here rather than run: an image whose source, made relative to the site, is not on it, by the site's base path and in each spelling a browser reads as another host, is dropped and reported. An image on the site is read as the path a browser requests, by the URL standard: the parser has decoded its character references and nothing decodes them again, tabs and line breaks are removed, a backslash is a slash, the path ends at a question mark or a number sign, and dot segments, written with periods or as %2e, are resolved. A path that leaves the site's base path, or names a parent directory only once decoded, is dropped and reported, and the file checked is the one the path names under the site's root, never one outside it, so that the report does not tell whether a file exists elsewhere on the server. Where that file is on the server running the conversion, it is checked with getimagesize(), as the filter checks it, and the image is kept with its relative source or dropped and reported. Where the file is not there, since that server may be a copy without the site's files, the image is kept and reported as unchecked when the path is under the public files directory or ends in an image file's extension, and is dropped otherwise. Amended 2026-10-31. Where the file is not there, the file a file resolver finds for the source is checked first, as 0041 records, and a source at an address the site was served at is the site's. The filter is what stops a page from making a reader's browser request an address on the site, such as /user/logout; this keeps that protection for every address but those two kinds, and accepts that a route answering at an address ending in .png, or under the public files directory, is requested by a page showing the document, rather than dropping every image of a site converted on a copy without its files. The site's host is the one the conversion runs under; where Drush runs without --uri, and its host is the placeholder default, an image at an absolute address is kept and that is reported, since dropping the site's own images would be the worse mistake. The filters whose work the converter does itself, filter_align, filter_caption, filter_htmlcorrector, filter_image_lazy_load and video_embed_wysiwyg, whose token a rule of its reads, or reports until a later wave does it, editor_file_reference, entity_links and media_embed, are not run. Amended 2026-10-31. The file and link resolvers of 0041 read what editor_file_reference and entity_links read. Any other enabled filter, and a source format that does not exist, is reported as unresolved in the report of every value converted from the format, as is an image kept because the site's host is not known, so that a report kept per value, as a migration keeps it, is complete; a run that adds up the reports of many values counts each of these once, because a filter such as one expanding tokens changes every value of the format alike.

Tables, disclosures and figures convert to the blocks that hold them. A table's rows are read in the order a page shows them and laid out on a grid by the HTML table model, a span reaching no further than its row group; cells written outside any row form one, as a browser makes it; a ragged row is filled with empty cells and overlapping spans are shortened, as the editor would on mount; a row holding no cell of its own is left out, and so is a column no cell starts in, which a browser gives no width. Spans are merged cells where the format permits them and are split otherwise. Header rows are the leading rows in a thead or holding only th, and header columns the leading positions holding th in every other row; both counts are then lowered until the editor reads them back unchanged, since a row below the header rows whose every cell is in a header column would itself become a header row. A cell holds lines: each paragraph, heading, quote, list item and code block in it is a line, a nested table's cells too, and any other block, such as an image or a media block with a caption, is moved after the table; the cell takes the alignment and colors its lines share. A caption becomes a paragraph before the table, and a table of one cell, or any table in a format without tables, has its content kept in its place. So does a table past a budget of grid slots the tables of one document share, as many as a document of the site's max_bytes holds empty cells: a slot is a row holding a cell of its own by a column a cell starts in, the layout reads a row from the cells reaching into it rather than slot by slot, and it gives up row by row, before anything the table's size is built. A table that passes a table limit the site sets, max_table_cells, max_table_columns or max_document_table_cells, counted as the document would hold it, keeps its content in its place too, since the gate would otherwise refuse the whole document for one table. A details is a toggle list item whose text is its summary, or a collapsible heading where the summary is one heading and the format permits one. A figure holding one image, video or audio element, no table, and its figcaption gives that element the figcaption as data-caption, the attribute Drupal's caption filter reads, and is unwrapped, so that whatever rule converts the element, the module's or a site's, takes the caption. Amended 2026-10-31. A rule that takes it into a block's inline content, as a media block holds its caption, removes data-caption from the element, and the figure rule reads that as the caption taken, as it reads a block's caption prop (0042). Each loss is reported.

The module's knowledge of common markup is rules on the same seam. Built-in rules registered at priority 0 map the classes common editors and word processors write (CommonClassMap), through a sixth class mapping, ClassMapping::ignore(), for a class that means nothing to a document and should not be reported; read video_embed_field's WYSIWYG token as the iframe it stood for (VideoEmbedTokenRule), so that whatever converts a provider's player converts the token; turn the paragraphs Word writes for a list into a list (WordListRule); and, when the caller asks with ConversionOptions::BREAKS_AS_PARAGRAPHS, read two or more line breaks in a row as the end of a paragraph (DoubleBreakRule), carrying the formatting around them into both halves through at most 32 inline elements, since cloning each of thousands of nested elements at every split took time in the cube of their depth. A site's rule at a higher priority comes first, and a site's class mapping wins over a built-in one. The options the module's own rules read are ConversionOptions constants, and drush blocknote:convert-html passes the one there is as --breaks-as-paragraphs.

The API is experimental until the converter has run on a real site's content, and docs/extending.md says so.

Alternatives considered¶

A Node script reusing upstream's parser. Rejected: it drops the attributes the conversion exists for, so it would need a pre-pass and a post-pass about the size of a converter of its own, a site without Node could not run it, and it would have neither the entity API nor the validator.

Rules as plugins. Rejected for the reasons above: a priority-ordered chain of services is core's own idiom, as for inbound path processors and breadcrumb builders.

Alter hooks or events around the conversion. Rejected: a second way to change what a conversion writes, with its own failure story, beside the rules.

Deciding block or inline from the element. The design classified an element as block-level when its registered rule declared that it produced blocks. Rejected after review: a custom element holding an encoded embed holds text, so it reads as phrasing content, and the rule interface would need a declaration it cannot make for every element; reading the result is exact.

Giving blocks their ids while walking. Rejected: a postprocessor could then add a block without an id, or duplicate one, which the validator refuses.

Gating with $entity->validate(). Rejected, above.

Keeping code and dropping what it meets, as ProseMirror's addToSet() and upstream's own parser do. Rejected for the reason 0037 gives: a link's target and a chosen color carry more than monospace does.

Matching an arbitrary color to the nearest palette color. Rejected: a silent change nobody chose. A site that wants a mapping writes a class mapping or a rule.

Parsing with Dom\HTMLDocument where PHP 8.4 offers it. Rejected: the same value would convert differently on two servers.

Mirroring filter_html from getHtmlRestrictions(), as the design proposed. Rejected after review: running core's process() is exact by definition, the escaped ampersands and the text of a removed tag included, and a mirror would drift from core.

Running filter_html_image_secure as it is. Rejected: it reads each image's file on the server running the conversion, which may be a copy without the site's files, so every image would be dropped there.

Keeping every image on the site without checking its file. Rejected after review: <img src="/user/logout"> would be kept, and a page showing the document would log its reader out, which is what the filter's check prevented.

Reporting a filter the converter does not run once per run. Rejected after review: a report kept per value, as a migration process plugin keeps it, would carry it only for the first value converted. A run that adds the reports up counts it once instead.

Laying a table out at any size. Rejected: a row of a thousand cells with rowspan="0" over a thousand rows, 41 KB of markup, made a grid of a million slots and took half a gigabyte, and the tables of one document repeat the cost. A table with more slots than a document holds empty cells fits in one only with most of its cells merged, so tying the budget to max_bytes costs a real table little and moves with a site that raises the limit.

Keeping a th wherever it is, or every header count the markup implies. Not possible: the editor holds header cells only in leading rows and leading positions, and recounts both when it writes a document back.

Converting video_embed_field's token into a link to the video. Rejected: the token stood for a player, and replacing it with the iframe it stood for lets the media submodule convert it as it converts any player, and the carrier keep it until then.

Reading a letter, or any number, as a list marker in a paragraph with Word's list class. Rejected after review: "A. Smith wrote" would become a numbered item, and "3.5 million" one starting at 5 without its number; a letter, a roman numeral or a number of several levels is a marker only where Word's mso-list style says the paragraph is in a list, and without it only 3., 3) and (3) are.

Matching classes by prefix, for the generated classes some sites copy, such as React Native Web's r-*. Not done: a site that meets them removes them with a DOM preprocessor, and a prefix match would be a second matching rule for every mapping.

Carrying markup that cannot be converted in a block of its own now. Deferred to the wave that writes documents into fields: with no write path, a dropped element is only a report line, and the carrier needs the editor to show it.

Consequences¶

A site converts with Drush; a migration process plugin and a batch form are planned after 1.0.0. Converting a field's every value, with revisions and translations, resolving files, content links and media items, and the carrier for markup that cannot be converted come with the waves the backlog item lists, and each adds to this seam rather than replacing it.

The fixtures in tests/fixtures/conversion/ are pinned against BlockNote 0.57.0's canonical shape, and an upgrade that changes it fails js/src/__tests__/conversion-roundtrip.test.ts, which opens every converted fixture in the editor through the client schema its format's pinned editor settings build, until the converter and the fixtures move with it.

Where masterminds/html5 builds a different tree from a browser, mostly around badly nested formatting in pasted Word markup, the document follows the parser and not the page. A CDATA section holding a > is one such place: a browser ends the comment it reads there at the first > and shows the rest of the section as text, and the converter removes the whole section. How often that matters is measured on a real site before the converter is finished.

The class report names a class the built-in mappings do not know once per class, counted; generated classes the module does not know, such as React Native Web's r-*, each have a line of their own until a site removes them.

propSchemaFor() is now getPropSchemaFor() and public; a third-party block that overrode it renames the method. A block implementing BlockNoteBlockInterface without extending BlockNoteBlockBase adds getEditorPropSchemaFor().

A document converted with the source format named differs from one converted without it wherever the format's filters changed what a page showed, and a run converting a field's values names it; drush blocknote:convert-html does with --source-format. A filter the converter does not run is reported, and a site that relies on one, such as a token filter, writes an HTML preprocessor doing its work, since the filters run after the preprocessors.

A table whose header cells are where the editor cannot hold them, a ragged table and a table with overlapping spans each convert to the nearest table the editor holds unchanged, and are reported.

References¶

src/Convert/ (HtmlConverter, HtmlWalker, BlockFactory, TargetPolicy, InlineNormalizer, StyleReader, SourceMarkup, SourceFormatFilters, TableGrid, ConversionGate, DocumentEncoder, NameBasedUuid, Report/, Rule/), src/BlockNoteBlockInterface.php, src/Drush/Commands/BlockNoteConvertCommands.php, blocknote.services.yml.
tests/modules/blocknote_convert_test, tests/src/Kernel/Convert/, tests/src/Unit/Convert/, tests/fixtures/conversion/, js/src/__tests__/conversion-roundtrip.test.ts.
Drupal core 11.4: Drupal\Component\Utility\Html::load(), Xss::filter(), FilterHtml::process(), FilterHtmlImageSecure::process(), ProcessedText::preRenderText(); masterminds/html5 2.10.
The HTML standard: the table processing model, the named character references and the character reference states of the tokenizer.
tiptap: the parseHTML() rules of @tiptap/extension-bold, -italic, -underline, -strike and -code. BlockNote 0.57.0: nodeToBlock.ts and blockToNode.ts, the table's tableContent, the block props' parseHTML. prosemirror-tables: fixTables().
docs/converting.md, docs/extending.md section 14.
0009, 0017, 0030, 0031, 0032, 0037.