Converting formatted text

This is what the HTML converter makes of formatted text, such as a body field written in CKEditor, what it leaves behind, and how a field's stored values are converted. Decision record 0038 says why the converter works the way it does, 0039 why a field is converted the way section 9 describes, 0042 why media is converted the way Media items describes, and docs/extending.md section 14 is the contract for a module that adds conversion rules of its own.

1. What exists today

The converter turns one piece of markup into one document for one BlockNote text format. It is a PHP service, blocknote.html_converter, which stores nothing; drush blocknote:convert-html, which runs it on a snippet; and drush blocknote:convert, which converts every value of a text field, in every translation of the revisions it is asked to, into the BlockNote field beside it, as section 9 describes. An image, a video or an audio element naming a file of the site becomes a block naming the file, and a link to content on the site a content link, as Files and content describes. With the media submodule, BlockNote Media, an image whose file a media item holds, an embedded media item and a YouTube or Vimeo player become media blocks, and items are created where a run asks, as Media items describes, those of videos whose media type fetches from the provider by drush blocknote:create-videos first, as Creating video items first describes.

The conversion API is experimental until the converter has run on a real site's content. It may change in any release before then.

2. Trying it

drush blocknote:convert-html body.html --format=blocknote --pretty
drush blocknote:convert-html --html='<p>Hello <b>world</b></p>' --report=json
cat body.html | drush blocknote:convert-html - --entity=node:12

The document is printed on standard output, exactly as it would be stored unless --pretty indents it, and the report on standard error, as a table, as JSON with --report=json, or not at all with --report=none. --format names the BlockNote format the document is meant for, blocknote when it is left out. --entity names the entity the markup belongs to, as entity_type:id, one in the trash included, and --target-field the field the document is meant for: a conversion rule may read either, and the block ids are derived from both, so that converting the same value again gives the same ids. --source-format names the format the markup was stored under, whose filters run first, as The source format describes, and which a conversion rule may read. --breaks-as-paragraphs reads two or more line breaks in a row as the end of a paragraph, as section 4 describes. --rule-option=name=value, which may be repeated, passes an option of a site's own conversion rules, such as mymodule.widget_field=field_widgets, as ConversionOptions holds it for them; the value is text, and the rule reads what it means. --unconverted=drop leaves out markup no rule can convert, which by default is kept as Markup that could not be converted describes. --site-url names an address the site was served at, such as a domain it has left, and --legacy-file-path a path its public files were once served at, such as /files, as Files and content describes; each names one, and is repeated for more. A comma may belong to an address or a path, so a value holding one is refused rather than split, and --site-url=a.example.com,b.example.com is a usage error. With the media submodule installed, --images, --create-media, --media-type and --media-owner say what becomes of images and players, as Media items describes; this command creates nothing, and reports each item a run would create as would_create.

One value cannot hold up a conversion for long. Markup larger than --max-source-bytes is refused before anything reads it, as source_too_large; by default the limit is four times the largest document the site stores, max_bytes in the blocknote.limits container parameter, 4 MiB at the default of 1 MiB, because markup is often larger than the document it becomes. A conversion that takes longer than --max-seconds, 10 by default, is refused as conversion_too_slow, checked between its stages and before each element is converted. Core's HTML parser cannot be stopped once it has started, and it takes time in the square of how deep elements nest, 1.5 seconds for 5000 nested div elements where it was measured, and the walk after it 3 seconds for 16,000 nested span elements each holding a letter, so markup it would take longer than the time left to read is refused before it is parsed, as markup_too_deep, by an estimate of its work, made each time the markup is about to be parsed: as stored, before the conversion rules and the source format's filters read it, before filter_html and the secure image filter parse what the filters before them left, before the converter parses the result, and before a rule parses markup of its own, as the media rule parses an image's data-caption. The estimate reads the markup with the parser's own tokenizer, keeps the open elements as its tree builder keeps them, and counts its walks up them and the depth it appends each node at; for filter_html it reads what Xss::filter() leaves, which is what that filter parses. An element carrying more than 256 attributes is refused there too, as markup_too_many_attributes, keyed by its name, since the parser sets them in time in the square of their number: 20,000 on one paragraph took it five seconds. The estimate counts those costs and no others: anything else the parser or a filter of the source format spends time on is refused once that stage has ended, as conversion_too_slow, as record 0039 explains. Either limit is turned off with 0. A refused value is reported, and the command exits with 1 as for any other refusal. Code calling the converter sets them with ConversionOptions::MAX_SOURCE_BYTES and MAX_SECONDS.

Exit code	Meaning
0	The document was converted and the checks a save runs accept it
1	The checks a save runs would refuse the document; the report's refused entries say why
2	The command was called wrongly, such as with a format that is not a BlockNote format, a file it cannot read or a --report value it does not know; nothing is printed on standard output
3	Something threw once the options were read, which is a defect in the converter, in a conversion rule or in what they load, such as a block plugin or the entity named by --entity; the message names the exception and where it was thrown

3. What converts to what

Source	Document
p, and phrasing content anywhere between blocks	paragraph
h1 to h6	heading, at the nearest level the format permits
ul, ol, li	bulletListItem and numberedListItem, the paragraph or heading an item starts with as its text and the rest nested under it; a nested list becomes the children of its item; an ordered list's start is kept from 1 to 1000000, the highest a page starts a list at, when the list starts with a numbered item of its own rather than a checkbox or a nested list
an li that starts with a checkbox, such as CKEditor 5's to-do list	checkListItem, ticked when the checkbox is
the paragraphs Word writes for a list, each with an mso-list style and its bullet or number as text, or with the MsoListParagraph class and its marker as text where a paste lost the style	bulletListItem and numberedListItem, nested by Word's level, numbered when the marker is a number, such as 3., 3) or (3), or, where the style is there, a number of several levels, such as 2.1., or a letter or a roman numeral followed by a period or a parenthesis, and starting at the marker's number; the marker is left out, and a paragraph with only the class that starts with other text, such as 3.5 million or A. Smith, stays a paragraph
blockquote	quote, the paragraphs it starts with joined by line breaks, anything after them nested under it
pre, with a language-* class on it or on a code inside it	codeBlock, its text kept exactly but for one newline at its start and one at its end, which a browser does not show, with the language when the format permits it; formatting inside it is left off, and a link keeps its text and is reported
hr	divider
img	image, naming the file the source names on the site, or with the source URL as written, as Files and content describes, the alternative text as its name, data-caption as its caption, data-align or align as its alignment and a width in pixels as its preview width, from 1 to 10000 pixels, the widest a preview is drawn; with the media submodule, drupalMedia where a media item holds the file or one is created, as Media items describes
video, audio, with a src or a source	video, audio, naming their file as an image does
div, section, article and the other containers	their content, which takes the container's alignment and colors
table, and CKEditor 5's figure class="table" around one	table, as Tables describes
details	toggleListItem, its summary as the text, a summary of several paragraphs as lines, any other block in it that holds text, such as a list item, as a line too, which is reported, and the rest nested under it; a heading that can be collapsed, at the summary's level, when the summary is one heading and the format permits a collapsible heading; one without a summary has no text, and is reported
figure holding one img, video or audio, no table, and its figcaption	the image, video or audio, with the figcaption's text as its caption, its line breaks and paragraphs read as spaces and a link in it reported, or a media block with the figcaption as its caption, formatting and links included; a figcaption no block takes, such as one whose image is dropped, stays a paragraph
any other figure	its content, the figcaption as a paragraph, and reported
drupal-media, and entity_embed's drupal-entity naming a media item	with the media submodule, drupalMedia, as Media items describes; otherwise kept as markup that could not be converted
an iframe holding a YouTube or a Vimeo player	with the media submodule, drupalMedia where a media item holds the video or one is created; otherwise kept as markup that could not be converted
any other iframe, object, embed, drupal-entity, svg, math, canvas and the like	kept as markup that could not be converted, as the next section describes
an element the converter does not know that holds nothing, such as a custom element an editor plugin stored	nothing, reported as dropped, since a browser shows nothing for it; a site's rule that knows what it stood for converts it or keeps it
a paragraph whose whole text is the token video_embed_field's WYSIWYG filter expands into a player, {"preview_thumbnail": …, "video_url": …, "settings": …}	an iframe loading the video's URL, which is then converted as any other player is; the URL is read whole where the source format's filter_url made a link of it and shortened its text; its thumbnail and player settings are not carried
script, style, template, noscript, document metadata and form controls	nothing, counted; a button's and a label's text is kept, as a page shows it

Classes are read only where they say something the document holds. text-align-left, -center, -right and -justify, which core's editor writes, and rteleft, rtecenter, rteright and rtejustify, which CKEditor 4's justify plugin was commonly set to write, are the block's alignment; CKEditor 5's image-style-* classes are an image's alignment, and its highlight classes marker-yellow, marker-green, marker-pink, marker-blue, pen-red and pen-green the palette color of the same name. The classes of CKEditor 5's to-do list and figures and its font sizes, Word's Mso* styles, LibreOffice's western, cjk and ctl, Safari's Apple-*, and notranslate, nanospell-typo and gtx-trans-icon, which translation and spelling tools leave, change nothing and are not reported. An element whose other classes are exactly a block style the target format lists for the block it becomes, in any order, gives that block the style, written as the format lists it (0046): under a format listing paragraph.lead.large|Lead, <p class="large lead"> becomes a paragraph carrying lead large, so a site whose CKEditor 5 format and BlockNote format list the same styles converts them with no mapping of its own. The element has to become that block itself, through a rule or a class mapping: an <li> gives its style to the list item it becomes, and a div whose text becomes a paragraph gives the paragraph none. A class mapping can map another class to a style, as docs/extending.md section 14 describes. Any other class is reported, so that a site can map it with a rule of its own; a class matching a style of another block type is reported too, once the document is built. The classes of an element that becomes no block, such as an empty paragraph, which a page showed nothing of, are not reported.

An image, a video or an audio element met in the middle of a paragraph splits the paragraph around it, because a document holds no image inside text; a small image in a sentence is reported for that reason, and a link around an image is lost and reported, because a media block carries no link. A block nested deeper than the site permits, 10 levels by default, is moved up beside its parent and reported.

A document has no list element, only list items, and the editor and the page number consecutive numbered items as one list. A numbered list right after another, with nothing visible between them, therefore goes on with the first list's numbers, and its own start is kept but not shown. Nothing is inserted to separate them, because anything a document could hold there would show on the page; each such list is reported as list_renumbered.

Markup that could not be converted

Markup the old page showed that no rule can convert, such as an embed code, a map or an svg drawing, is kept as it was, in a block of its own type, blocknote_unconverted, whose html prop holds the element's markup as the converter met it and whose reason prop holds the report code that says why, such as embed_not_converted. Elements of this kind that follow one another, with nothing but whitespace between them, share one block, its reasons separated by spaces, because they are one thing for an author to rebuild. An element met in the middle of a paragraph splits it, as an image does. Each element is reported as carried.

No plugin provides the block, so the page leaves it out, as it leaves out any block no module provides, and its markup never reaches a reader. The editor shows it as Markup that could not be converted, with the reason in words and the markup as text, never as markup, and the document cannot be saved while it holds one: the author rebuilds what it shows with the editor's own blocks, such as a media item for a video, and removes it. The status report counts the documents in current revisions that still hold one, on a line of its own. Until then the page shows what it would show had the markup been dropped, but the markup is counted and recoverable rather than lost. 0040 records why.

Markup the old page never showed, such as a script, a style or a form control, is not kept, and neither is markup a rule drops on purpose. A conversion that names ConversionOptions::UNCONVERTED as drop, as --unconverted=drop does, drops what would have been kept and reports it as dropped, and so does a conversion for a format that holds one paragraph, which holds no other block.

Files and content

A document names a file of the site by its identity, file:<uuid>, so that it follows the file wherever the file moves, and content by a content link, which follows the content wherever its address moves (0028). The converter finds both through resolvers, conversion rules a site may add to (docs/extending.md section 14), and 0041 records why.

An address is the site's when it is relative to the site's root, such as /node/12, or absolute at the host the conversion runs under, which Drush's --uri gives, or at an address --site-url names, such as https://www.example.com or a domain, example.com, with the path the site was served under there if it was not its root; http and https are the same, and where an address is under two paths named at one host, the longest wins. It is read as a browser requests it, its dot segments resolved and its path ending at a ? or a #, and an address that leads out of the site is not the site's. An address relative to the page, such as page.html or #top, names nothing.

An image, a video or an audio element names a file by the data-entity-uuid core's editor wrote with data-entity-type="file", which wins, since the filter that showed the page used the file with that uuid, or by its source on the site: under the directory the public files are served from, /sites/default/files/ on most sites, or under a path --legacy-file-path names, which stands for it too. The uuid is read only where the source format runs editor_file_reference, or the conversion names no source format, since otherwise the page showed the source as written. The attributes are read as core's editor module reads them, exactly as written, and the uuid names a file only when it is that file's own, in lower case: data-entity-type="File", or a uuid in capitals or with a space around it, names no file, and the source is read instead. The file named by its source is the one with that URI, the first saved where two have it. An image style's derivative, such as /sites/default/files/styles/large/public/a.png?itok=…, names the public file it was made from, its itok left aside, and the style is reported, since the block shows the file itself at its own preview width. The block names the file as file:<uuid>. A source no file is found for is kept as written, and reported when the element named a uuid or the source is on the site; a source elsewhere is an address and is not reported.

Where the format permits content links, a link names content by the data-entity-type and data-entity-uuid core's entity link plugin and linkit write, where the source format runs core's entity_links filter or linkit's, or the conversion names no source format, read exactly as written, as those filters read them, or by its address on the site: a file of the site, other than a derivative, since a link to a file is the file, or the canonical address of content, such as /node/12, read as the router reads it, so that /node/12.0 names nothing, or an alias of one, looked up in the language a path prefix names where the site negotiates the language by path prefix. Where it negotiates the language by domain, no alias is looked up, since an alias names content in the language of the host a page was served at, which the address does not say: a link through an alias in any language stays plain and is reported, and a site whose links need it writes a resolver of its own that reads its domains. Any other page a route answers, such as a view or the front page, is no content, and the link to it stays a plain link without a word. An address of the site that names nothing stays a plain link and is reported, once every link resolver, a site's own whatever its priority included, has found nothing at it: as link_content_not_found when it is the canonical address of content the site no longer holds, such as /node/12 for a deleted page, also through an alias, and as link_address_not_answered when nothing answers it, such as /articles/foo of a site since replaced, keyed by its path once the language prefix is taken off and the alias resolved. Something answers an address when, once the language prefix and the alias are taken off, a route matches it as the router matches it, or a file or directory of that name is in the docroot. A redirect, such as the redirect module's, is not asked, and a route that matches answers whatever its parameters name, so /node/12/edit is answered whether or not node 12 exists. An address in a files directory, whether a file is there or not, is the file rules' and is not reported here, and neither is one under /system/files/, where core serves the private files, nor a link that named content by its attributes, which entity_link_kept_as_url reports, nor one a resolver of the site found. A path is looked up once in a run, and only where the format permits content links, since the link resolvers are asked only then. A link to content stays a plain link too, and is reported, when it says what a content link cannot hold: a query or a fragment, such as /node/12#comments, or a translation in another language than the document's. A content link leads to the content in the language of the page showing it, so a link that led every page to one translation leads each page to its own once it is a content link. On a site with language path prefixes, a link through an alias written without a prefix is such a link where every language has a prefix: the address names no language, its alias is looked up in the default language, and it becomes a content link in a document in any language. linkit's data-entity-substitution is kept where the format's content link settings permit it, and otherwise left off, so that the content link follows the base rule, and reported. In a format that does not permit content links every link stays plain, and one naming content by its attributes is reported. The text of a content link keeps its formatting and its new-tab flag, as a link's does.

A document naming a file opens it to whoever may view the document: the file gets a usage row, and the module serves a file a document names to the document's readers. A file in the public files directory is served to anyone already. Any other, such as a private file, is named only where core's editor module recorded the entity the markup was read from as using it, which it does when the entity's formatted text names the file with exactly data-entity-type="file" and its uuid, and which serves the file to whoever may view that entity already. A file the document may not name, however it was found, a site's own resolver included, is kept as its address, the image's source or the link's, and reported as private_file_kept_as_url; so is every such file in markup converted without an entity, as drush blocknote:convert-html converts it without --entity. A private file is never found by its address, such as /system/files/…, which guarded the file by its own access, nor by the address of an image style's derivative of it. A document a conversion rule built that names a file it may not is refused as private_file_named.

Content is found whoever runs the conversion: an unpublished page, or one the account running Drush may not see, becomes a content link like any other, and the page decides for each reader, as a page showing a plain link to it did. A link to a file, though, shows its text without an address to a reader the file's access refuses, where the old link led to a refusal.

Each revision a run writes gives every file its documents name a usage row, as a file uploaded in the editor has, and makes a temporary file permanent. A site's own address patterns, such as those of the site it migrated from that no route of the new site answers, are a resolver of its own; docs/extending.md section 14 shows one.

A file in the public files directory that no file entity names, such as one copied there by hand or by a migration that wrote no file record, keeps its address unless the conversion registers files, as --register-files and ConversionOptions::REGISTER_FILES ask. An image, a video or an audio element whose source is such a file, or an image style's derivative of one, then names a file entity created for it, when no resolver, a site's own included, found one: its uuid is derived from the file's URI, so that every document of a run naming the file names one entity, it is permanent and owned by the owner of the entity the markup belongs to, and it is created only when the document naming it is written, in the same transaction, as a media item is (Media items). Each such element is reported as would_create, keyed file.file, and created once the write has created it. The file must be on the server running the conversion, since a file entity for a file that is not there would name nothing, and a link to such a file stays a plain link, as before. A file is registered only where it is plainly a public file the element shows: the source's path may not encode a slash or a backslash, which a browser reads as part of a name and a server as a separator, nor name a dot segment; the file, its links followed, must be inside the public files directory, compared by real path, and outside the private and the temporary directories where the public one holds them; it may not be hidden, its name or a directory's starting with a period, nor be in a directory Drupal writes files of its own into, styles, css, js, php or translations at the top of the public files; its extension must be an image's for an image (avif, bmp, gif, ico, jpeg, jpg, png, svg, tif, tiff, webp), a video's for a video (m4v, mov, mp4, ogg, ogv, webm) and an audio file's for an audio element (aac, flac, m4a, mp3, oga, ogg, opus, wav, webm), in either case; and its URI must fit the 255 characters a file entity holds. Where no entity is registered, for one of these reasons, because the file is not on the server, the revision is an older one, or the markup belongs to no entity, which has no owner for the file, as in drush blocknote:convert-html without --entity, the source is kept and reported as file_reference_kept_as_url as before, its detail saying why. With the media submodule, an image whose file is registered this way stays an image block in that run, since no media item holds a file that is not saved yet; a run with --create-media after it finds the file and makes a media item of it.

Media items

With the media submodule, BlockNote Media, installed, and where the target format enables its media block, markup standing for a media item becomes a drupalMedia block naming the item by its uuid, and the media types the format permits are the only ones used. 0042 records why it works this way, and docs/extending.md section 14 how a site's own rules use it.

An item is found first, by media resolvers, conversion rules a site may add to. The module's own finds an item by its uuid, an image's item by the file its image field holds, and a video's item by the address its remote video field holds, whatever form of the video's address that is; where several items hold the same, a published one is found before one that is not, and the first saved before the others. Finding reads only the site's own data, and asks no access: the run's account may see what a reader may not, and the page decides access to an item for each reader.

A page showed an image or a player to each of its readers, and a media block shows an unpublished item only to those who may view unpublished media, so an item found by the file or the video it holds is named only where it is published, or where the items the run creates for the document would be unpublished too, as for a draft. An older revision saved as the default one was the page its readers saw, and is again once it is reverted to, so it names such an item as it did while it was current, and converting it again once it is older writes the same document. Otherwise the image stays an image block and the player is kept, reported as media_not_published, keyed by the item's media type, and no other item is created in its place; publish the item and run again with --overwrite. The exception is an item the running job created unpublished, for a draft or another page its readers did not see: where the conversion creates items, a page its readers see that names the same image or video later in the same job, in the same run or in a run resuming it, names the item, and its write publishes it, reported as would_publish and then published, keyed by the media type. A job is given an id when it starts, its key and a uuid, which every run resuming it keeps and --restart replaces, and an item it creates unpublished has the revision log message "Created unpublished by the BlockNote converter in job" and the id; one it creates published has "Created by the BlockNote converter in job" and the id. An item is published only where its log is the unpublished one of the running job, its uuid is the one the conversion derives for what it holds and its only revision is the one the job saved, and the write checks all of it again on the item as it is then: where the item no longer passes, or is in the trash, the write publishes nothing and leaves the revision as it was, as skipped_changed_during_run, and the next run reports the item as media_not_published. Cron's thumbnail and title queues save an item in place and leave its log as it is. The media form empties the log, and saving a revision of an item adds one, so an item someone edited stays unpublished; so does an item an earlier job created, and one the job created published that someone unpublished, with core's action too, which saves in place and keeps the log. An item whose media type is under content moderation is never published this way, since its workflow sets its status on every save: it is reported as media_not_published, its detail naming moderation, and moves through its workflow as any other item does. What the check cannot tell is an item the job created unpublished that someone means to keep unpublished, or unpublishes in place after the job published it, while the job still runs: the job publishes it, so unpublish such items once the job is done. drush blocknote:convert-html and a dry run, which is a job of its own, publish nothing, and report such an item as media_not_published; a dry run cannot foresee an item an earlier document of the same run would create either, and reports would_create for the later page as for any item it plans. An embedded item is named whatever its status, as the page named it.

• Images. --images=auto, the default, makes an image whose file of the site an item holds a media block, and any other image an image block, as before; --images=image keeps every image an image block; --images=media does as auto and reports each image that stays an image block, once: as media_not_created, with the reason, where an item was to be created for it, and as media_not_found otherwise. The file is found as Files and content says. The image's caption, from data-caption or a figure's figcaption, becomes the block's caption with its formatting and links, its line breaks and the edges of its paragraphs read as spaces, as an image block's are, and data-align or align its alignment. Its width is not carried, since the view mode decides the size, and is counted as media_width_not_carried.
• Alternative text. Where the item's image field has alternative text enabled, an image's alt that differs from the item's own overrides it for this use, an empty alt marks the image decorative, and an image without alt shows the item's own, as an author sets them in the editor; text longer than the 512 characters a block holds is cut and reported as alt_shortened.
• Embedded items. Core's <drupal-media> is read only where the source format runs media_embed, and entity_embed's <drupal-entity data-entity-type="media"> only where it runs entity_embed, since a page showed nothing for either elsewhere; such an element is dropped and reported as embed_not_filtered, and so is one naming no item by a uuid, as media_reference_invalid. entity_embed's element is read as its filter read it: the item is the one data-entity-uuid names, or, where the element names no uuid, the one data-entity-id names, and an element naming neither a display nor a view mode, which the filter did not show, is dropped as media_reference_invalid; an id no item has is reported as media_not_found, and the element left out, since no block can name it. The view mode is data-view-mode, or the source filter's default view mode, or for entity_embed's element the view mode its display names, or its data-view-mode where it names no display and no display settings, as entity_embed still reads it; any other display is the default view mode, reported as embed_display_not_carried. data-align and data-caption are read as for an image, and alt, where "" marks the image decorative. An item the site does not hold is named by its uuid, so that it shows once the item exists, and reported as media_not_found; an item of a type the format does not permit is kept as markup that could not be converted, and reported as media_type_not_permitted.
• Players. An iframe holding a YouTube or a Vimeo player becomes the item holding its video, and video_embed_field's token does, once it is a player. The address is read in any of its forms: YouTube's watch, embed, shorts, live and v addresses, youtu.be, also at www. and m., and the privacy-enhanced youtube-nocookie.com, and Vimeo's pages, channels, groups, albums, showcases and player, at the scheme's own port where one is written. An unlisted Vimeo video's hash, read from h= or from the segment after the id in a page's address, where it has the ten lower-case hexadecimal digits of Vimeo's hashes, is kept, in the item's address and in what the video is known by, since Vimeo shows the video only with it; a segment of another shape is a page about the video, not a player, and an address naming two hashes names no video: an item holding the video's address without the hash is not the item for a player holding it, nor the other way round. A player's settings are not carried; its start time is reported as video_start_not_carried, its end time as video_end_not_carried, a playlist it played the video in as playlist_not_carried, and the privacy-enhanced player as privacy_mode_not_carried, since the item shows the provider's own player, alone, from its start to its end. A player no item holds is kept as markup that could not be converted and reported as media_not_found, unless an item is created for it. A playlist, a channel and a player of another provider are kept as before.
• View modes. A view mode the format does not permit is the default one, reported as view_mode_not_permitted.
• Galleries. No markup becomes a gallery by itself, since there is no common gallery markup to read: each site writes its own, often a placeholder only its own code understands. A site's rule that recognizes it builds the gallery from the items with one call, as docs/extending.md section 14 describes; where the target format does not enable the gallery, each item becomes a media block of its own, reported as block_not_permitted.

Creating items. --create-media creates an item for an image's file or a video that no item holds, in the first media type --media-type names that holds it: one of core's image source for an image, and one with an oEmbed source permitting the provider for a video, such as core's Remote video. --media-type is repeated for each type, in the order the types are tried, since an item is created in the first that holds it, and each must be a type the target format permits: both commands refuse another as a usage error, against --format, or for drush blocknote:convert the one format the target field allows where --format is left out. The item is owned by the user --media-owner names, by id, or else by the owner of the entity the markup belongs to, in the language of the translation it was converted from. The values a site's rule sets on it are validated as that owner, not as the account running the conversion, which for Drush is anonymous: a formatted text value is set only in a format the owner may use, so the owner, and where --media-owner is left out each host's owner, has to be able to use the formats of the fields the values are set in, or the value is left off and reported as media_value_not_set. The item is published only where the page naming it is: for the default revision of a translation that is published or has no published status and is not in the trash, and for markup belonging to no entity. An item created for a draft, for an unpublished translation or for one in the trash of the Trash module is unpublished, so that it shows no one the page did not show; publish it with its page. The older pass creates no item, since a page no longer shows an older revision: its image stays an image block unless an item holds the file. An image's item holds the image's alternative text, empty for an image whose alt is empty, which marks it decorative; where the media type's image field requires alternative text, no item is created for an image without an alt attribute, which stays an image block. A video's item holds the video's page address and is named by it. No item is created for a file outside the public files directory, since a new item would serve the file to whoever may view the item.

An item is created only when the document naming it is written, in the same transaction, so a dry run, drush blocknote:convert-html, a refused document and a document left as it was create nothing, and neither does an image whose block the document does not keep, such as one in a caption, which a caption cannot hold. Each element naming an item its document's write creates is reported as would_create, in the category created, keyed by the media type, such as media.image; once the write has created it, the entry says created, and the documents of the run that name an item another document's write created say nothing of it, as for any item found. Only a document that is written, or would be in a dry run, says either: a document refused, or skipped because its field holds another, says nothing of what its write would create or publish, and the created column of documents.csv counts only what is created, and what is published. A document the converter wrote that converts the same again is left as it was, unless it names an item that is gone, such as one deleted since: it is written again, which creates the item again with its uuid. An item in the trash of the Trash module is not gone, since its uuid is still taken, and no page shows it, so it is neither named nor created again: the image stays an image block, reported as media_not_created keyed trashed, and a document the converter wrote naming the item differs from the conversion, which --overwrite writes, until the item is restored or deleted for good. Every document naming the same image or video names one item, whose uuid is derived from its media type and what it holds, so a run after the site's database is reset creates it again with the same uuid, and a second run finds it. An item that has that uuid but is of another type, or holds another file or video, is never named: the element is converted as without creation, reported as media_not_created keyed uuid_taken, and where such an item, or one in the trash, appears while the run converts a revision, the revision is left as it was and skipped as changed during the run.

A conversion fetches nothing from the network, and neither does creating an item, but for the oEmbed provider list, below. A remote video type is used for creating items only where saving one asks the provider for nothing: tick Queue thumbnail downloads on the type, and map no metadata into a field but the name, as core's Remote video type maps the title. Cron fetches each thumbnail afterwards. The command refuses another type as a usage error, and a site whose remote video type fetches, as core's Remote video type does until it is changed, creates its video items first, as Creating video items first describes. A video's item is named by its address, and cron fetches its title afterwards too, through the queue BlockNote video title fetcher, and renames the item where its name is still the address, so a name someone gave it meanwhile is kept; an item the provider gives no title for keeps its address. A provider that throttles the queue stops it until the next cron run, which asks for the same item again; a timeout, a failed connection or a server error queues the item again, up to three tries in all, after which it keeps its address. Where creation was asked and does not happen, the element is converted as without it, and the reason is reported as media_not_created, keyed owner, private_file, type when no named type holds it, type_not_permitted, type_fetches, address_too_long when the address is longer than the type's source field holds, address_not_provider, older_revision, uuid_taken, trashed or alt_missing. An item's name longer than its name field holds, such as a long address, is cut to fit.

An address of another provider than YouTube and Vimeo, which a site's rule names the provider of, gets an item only where one of that provider's URL schemes, in core's oEmbed provider list, matches it; otherwise it is reported as media_not_created, keyed address_not_provider, since core fetches an address no scheme matches to discover its provider when it renders the item or fetches its thumbnail. The list is read through core's provider repository, which fetches it from the address the site configures, oembed.com by default, once its stored copy is a week old: the one request a conversion may make, and never to an address the markup holds. The title queue checks each item's address the same way before it fetches anything, and that the provider is one the item's media type allows, and leaves any other named by its address.

Creating video items first

drush blocknote:create-videos <entity_type> <source_field> <target_field> --media-type=remote_video creates the remote video items of a field's conversion before it, for a site whose remote video type fetches the video's title and thumbnail from the provider as an item is saved. A conversion cannot create those, since it fetches nothing from the network while it writes, so this is the one place the module asks a provider for anything, and it does so as core does when an author adds the item: by saving the item, which fetches through core's oEmbed resource fetcher. 0042 records the decision.

It reads the values drush blocknote:convert would convert, with the same entity type, fields, --bundle, --format, --ids and --rule-option and the same rules, converting each current revision and writing nothing, and takes from the conversion the videos it would create an item for: those no item holds, in any form of their address, once each however many values name them. It does not read the dry run's report, which names a video by a sentence and an excerpt of the markup, names none at all when the type fetches, and says what the content was when the dry run ran. It reads the values again each time it runs, which costs a dry run, and fetches nothing to do it: --dry-run lists the videos that would get an item, and the type each would be created in, and asks no provider.

--media-type is repeated for each type, in the order they are tried, and each must have an oEmbed source permitting the video's provider and be permitted by the target format, since the conversion finds an item only among the types the format permits. --media-owner owns the items created, else the owner of the entity the video is embedded in; an item is published where a published page names the video, and unpublished where only a draft, an unpublished translation or a page in the trash does, as for an item a conversion creates. The older revisions are not read, since they create no item.

Each video is asked for from its provider before its item is saved, so that a video the provider refuses, or that cannot be reached, is named with the reason, and gets no item, where core would save one named by a placeholder and say nothing; the command goes on with the next video, ends with exit code 1 and prints failed <address>: <reason> for each. Running it again creates only the items still missing: an address an item holds is skipped, whatever made the item, so a run stopped by Ctrl-C, by --limit, which creates at most that many items, or by a provider throttling the requests is resumed by running it again. --batch-size is how many entities it reads at a time. The command prints created <address> as <uuid> in <type> for each item.

Run it before the rehearsal, with the same selection, fields and --rule-option as the conversion, and the dry run of drush blocknote:convert then finds the items and reports no video as would_create. The conversion itself needs no --media-type for videos, only for images, --create-media --media-type=image, and still fetches nothing.

Tables

A table becomes a table block, its rows in the order a page shows them: the first thead's at the top and the first tfoot's at the bottom, wherever they are written. Its cells are laid out on a grid as a browser lays them out. A row shorter than the table is filled with empty cells, as the editor fills it, and a row holding no cell of its own, because the spans above cover it or it is empty, is left out, as a page shows nothing for it. A span reaches no further than its thead, tbody or tfoot, as in a browser; two spans that would overlap, which a browser draws on top of each other, are made smaller; and a column only a span reaches into, such as those colspan="100" makes past a table's own columns, is left out, as a browser gives it no width. Cells written outside any tr form a row of their own, as a browser makes one for them.

colspan and rowspan become merged cells where the format permits them. Where it does not, each merged cell keeps its content in its first slot and the slots it covered become empty cells.

The leading rows that are in a thead, or hold only th cells, are header rows, and the leading columns whose cells are th in every other row are header columns, where the format permits header cells. A header row shorter than the table is still one, the empty cells it is filled with becoming headers too. The editor holds headers only there, so a th anywhere else is an ordinary cell. The editor also counts a row as a header row whenever every one of its cells is in a header column, so where a row below the header rows is that short, as a row of one cell spanning the table is, there are fewer header columns, or none.

A cell holds lines of text. Each paragraph in it is a line, and so is a heading, a quote, a list item, a code block or another table's cells; each loses what made it that block. Any other block, such as an image, or a media block, whose content is its caption, is moved after the table. A cell takes the alignment and the colors its lines share, which they inherit from the cell's own style and align, from its row and its thead, tbody or tfoot, and from the table and what holds it, and an empty cell takes them from there itself; a cell's colors need the format to permit cell colors. A column takes the width in pixels a col or a colgroup gives it, in a width attribute or declaration, or else the width of the first cell that starts in it and spans it alone, from 1 to 10000 pixels; a width in another unit, such as the percentages CKEditor 5 writes, is not carried. The align attribute of a table places the table and is not its text's alignment.

A caption becomes a paragraph before the table, where a page shows it, and so does anything a page moves out of the table, such as text written between its rows. A table of one cell lays out what it holds, so that content is kept in its place, without the table. In a format that does not permit tables, every table's content is kept in its place the same way, as paragraphs.

The tables of one document are laid out on a grid of at most as many slots, counted together, as a document of the site's largest size holds empty cells: max_bytes in the blocknote.limits container parameter divided by the 141 bytes an empty cell takes, 7,436 at the default of 1 MiB. A slot is a row holding a cell of its own by a column a cell starts in, so the rows and columns a page leaves out cost nothing. A table that would pass the budget, as a row of cells with rowspan="0" over as many rows does in a few kilobytes, keeps its content in its place as paragraphs, and is reported as table_too_large. Held as a table, it would be larger than a document may be unless most of its cells were merged. Where the site also sets a table limit in that parameter, max_table_cells, max_table_columns or max_document_table_cells, a table that would pass one, counted as the document holds it, keeps its content in its place the same way and is reported the same, since the checks a save runs would refuse the whole document for it: a merged cell is one cell, the empty cells a row is filled with count, and the cells of a table inside a cell, which become lines, do not.

Callouts

The converter recognizes no callout markup, because every site marks its callouts its own way, with classes such as highlighted, alert, callout or note, and a guess from those would convert what is no callout on some site (0048). With BlockNote Extras installed and the target format enabling the callout, a site maps its own class to one with a class mapping, as docs/extending.md section 14 describes, in one line for each class:

ClassMapping::block('p', 'highlighted', 'callout', ['tone' => 'info']),

The mapped element's text becomes the callout's, and the blocks inside it the callout's children, the paragraphs it starts with joined as its text. A callout holds no heading, so BlockNote Extras keeps every callout a conversion builds free of them: a heading the callout starts with, before any text of its own, becomes its title where it has none, every other heading inside it, at any depth, a paragraph keeping its text, its colors, its alignment and its children, and a callout left with no text of its own takes the paragraphs it then starts with as its text. Each heading is reported, and so is the id of such a heading, as heading_id_not_carried, since it is no heading on the page. Where the target format does not enable the callout, the mapped element becomes a paragraph, as any block the format does not enable does.

The source format

A page showed the stored markup after the filters of the format it was stored under had changed it. When the conversion names that format, as --source-format does, and as drush blocknote:convert does for each value, naming the site's fallback format, plain text unless the site changed it, for a value stored without one, since core shows such a value in it, its enabled filters that change what a page shows of plain markup run on the markup before it is read, in the format's weight order and with the format's own settings: filter_html_escape, which makes the whole value text, filter_autop, which makes paragraphs and line breaks of its line ends, filter_url, which makes links of addresses, and filter_html, as core runs it, so that a tag or an attribute the format never let a page show is not converted either, though the text inside such a tag is, as a page showed it. filter_html_image_secure shows only an image file from the site, at the address relative to the site it gives the page, and puts an error icon in place of any other. So an image whose source is elsewhere is dropped and reported. An image on the site is read as the path a browser requests: the parser has read its character references, so ? is the text ? and nothing more, its tabs and line breaks are removed, a backslash is a slash, the path ends at a ? or a #, and its dot segments, written with periods or as %2e, are resolved. A path a browser resolves outside the site's base path, or one that names a parent directory only once it is decoded, as %2f.. does, is dropped and reported. The file checked is the one the path names under the site's root, never one outside it. When that file is on the server running the conversion, the image is kept with its relative address when the file is an image, and dropped and reported when it is not, such as /user/logout. Where the file is not on that server, as on a copy of a site without its files, the file the resolvers of Files and content find for the source is checked instead, the file an image style's derivative was made from for a derivative: where that file is on the server, as the filter checks it, and where it is not, one the site records as no image, such as a PDF, is dropped and reported. Otherwise an image under the public files directory or whose address ends in an image file's extension, such as .png or .jpg, is kept and reported as unchecked, and any other is dropped and reported. The filter read a query as part of the file's name, so it showed an error icon for every image style derivative with its itok; the converter keeps such an image where its file, or the file it was made from, is an image. Whether an absolute address is the site's own is decided by the host the conversion runs under and the addresses --site-url names, so run Drush with --uri set to the site's address; without it, an image at an absolute address no site URL names is kept, and that is reported once for each value that holds one.

The filters whose work the converter does itself are not run: filter_align and filter_caption, whose data-align and data-caption it reads, filter_htmlcorrector, as it repairs markup the way a browser does, filter_image_lazy_load, which changes nothing a reader sees, editor_file_reference and entity_links, and linkit's filter, whose data-entity-* attributes it reads as Files and content describes, and only where the format enables the filter that read them: where it does not, the page showed the address as written, so the converter reads the address and reports the attributes as entity_attributes_not_filtered, media_embed, whose drupal-media elements the media submodule reads, as Media items describes, and which are kept as markup that could not be converted without it, and video_embed_wysiwyg, whose token it reads as section 3 says. entity_embed's entity_embed is not run either, and the media submodule reads its drupal-entity elements that name a media item. Any other enabled filter, such as one that expands a token, is reported, and so is a format that does not exist, in the report of every value converted from the format, so that each value's report is complete on its own. A run that adds up the reports of many values counts each of these once, because it says the same of every value of the format. A conversion that names no format reads the markup as stored.

4. Inline content

Tags and style declarations become formatting the way the editor's own paste reads them: b and strong, a font-weight of bold, bolder or 500 and above, i and em, font-style: italic, u and text-decoration: underline, s, del, strike and text-decoration: line-through, code, and sub, sup and vertical-align. A <mark> with no class and no background of its own, which the paste leaves as text, becomes the yellow background a browser draws it in, where the format permits it; one with a class, such as CKEditor 5's highlights, takes the color its class says. A font-weight of 400 and font-style: normal take bold and italic off the text inside. A <b> that declares font-weight: normal, which Google Docs wraps around everything it copies, is not bold.

A color converts only when it is one of the nine palette names, gray (in either spelling), brown, red, orange, yellow, green, blue, purple and pink, written in a color or background-color declaration, a <font color> or a data-text-color or data-background-color attribute. Any other color is reported, because a color guessed from a hexadecimal value would be a change nobody chose. On a block element a color is the block's own; on text it is the text's.

A link keeps its target and its text. target="_blank" becomes the link's new-tab flag. Inside a link this module rendered with the blocknote-link--new-tab class, an element with the visually-hidden class, the hidden text that says the link opens in a new tab, is left out, since the flag says it again; the element is reported as new_tab_text_removed, and a visually-hidden element anywhere else is converted as any other. A link inside a link is lost and reported, its text kept in the outer one. A link to content on the site becomes a content link where the format permits one, as Files and content describes.

Whitespace collapses as a browser shows it, except that a line break in text whose white-space declaration keeps line breaks, pre, pre-wrap, pre-line or break-spaces, such as the pre-wrap Google Docs puts on every run it copies, is a line break, as a <br> is. A single no-break space between two words stays one; two or more, or one beside a space, become one space; whitespace at the start and the end of a block and beside a line break goes. <br> is a line break, except at the end of a block, where a browser draws nothing for it. Older content often separates paragraphs with two line breaks rather than paragraph elements, and the conversion option breaks_as_paragraphs (ConversionOptions::BREAKS_AS_PARAGRAPHS, or --breaks-as-paragraphs) reads two or more in a row, with nothing but whitespace between them, as the end of a paragraph: a paragraph is split in two there, its formatting carried into both halves through the 32 inline elements nearest the paragraph, the formatting of any further in staying with the text before the line breaks and reported, and the content on either side of them anywhere else becomes two paragraphs, which a quote, a heading and a table cell then hold as their lines, and a list item as its text and a paragraph nested under it. It is off by default, because the same markup also spaces lines apart inside one paragraph, such as the stanzas of a poem. A paragraph holding nothing visible, such as <p>&nbsp;</p> or <p><br></p>, is left out and counted.

Character references are read as a browser reads them. A numeric reference to 0, to a surrogate or past the last Unicode character is U+FFFD, and one to a number from 128 to 159 is the Windows-1252 character a page in that encoding meant, such as ’ for ’. The 106 names a browser reads without a semicolon, such as &copy and &AMP, are read without one, the longest such name a word starts with, so that &notit; is ¬it;, except in an attribute when a letter, a digit or = follows, so that a link to ?a=1&copy=2 keeps its address. A control character, such as U+0001, written as itself or as a reference, is removed and counted, because a browser draws no letter for it; tab, line feed, form feed and carriage return are whitespace. Bytes that are not UTF-8 become U+FFFD and are reported, whether they are in the stored markup, in markup a conversion rule returns or in text a rule writes. A CDATA section, <![CDATA[...]]>, is left out, because a browser reads it as a comment. A browser ends that comment at the first >, though, so where a section holds one, the page showed the rest of it as text, such as y]]> from <![CDATA[x>y]]>, and the document leaves that out too.

5. What the target format decides

The converter consults the format while it builds, so that it writes nothing the format refuses, and reports each narrowing:

• A block the format does not enable becomes a paragraph holding its text when it holds text, such as a heading, a quote, a list item or a code block, and is dropped otherwise, such as a divider or an image.
• A heading level the format does not permit becomes the nearest one it does; of two as near, the less prominent.
• A style, a color, an alignment or a code language the format does not permit is left off, and the text keeps the rest.
• A block takes a block style only where the format lists it for the block's type, so a style a class mapping names that the type does not list is left off and reported.
• A link the format does not permit, or whose scheme it does not permit, leaves its text without the link. A new-tab flag the format does not permit is left off.
• Code formatting cannot share text with any other formatting, a color, the new-tab flag or a link, and subscript cannot share it with superscript, because the editor cannot show those together (0037). The converter keeps the other formatting and leaves off the code, or the subscript.
• In a format that holds one paragraph (0030), every block's text is joined into one paragraph, with a line break between them where the format permits one and a space where it does not.

Last, the document goes through the checks a save runs, including the site's size limits. A document they refuse is reported as refused and must not be stored; for anything but a size limit, that is a defect in the converter or in a conversion rule.

6. What is not carried

A document holds what the editor can edit, so these are not carried: style attributes beyond the formatting, colors and alignment above, classes other than a block style's and ids, a heading's included, since a page gives every heading an id made from its words (docs/rendering.md section 5), data-* attributes other than those named above, title, rel, hreflang and class on links, reversed and type on lists and value on list items, an image's height, a media block's width, a player's settings, start and end times, playlist and privacy-enhanced host, an embedded item's language and title override, a table's borders, spacing and placement and column widths in a unit other than pixels, whether a details was open, font families and sizes, margins and indentation, and the visual quotation marks of <q>. Classes, list attributes, font families and colors outside the palette are reported, and so is a heading's id when the page will not give the heading the same one, and so is an <a>'s id or name, a place a link with a fragment reached, unless the page gives a heading the same id; the rest are not, because they would fill the report and nothing in the document could hold them.

7. The report

Every conversion has a report. Each entry has a category, a code, a key saying what it concerns, such as a tag, a class or a value, the path in the markup where it was first met, such as body/div[2]/p[3], or body/x-embed[2]/iframe[1] inside the markup a rule put in the place of the second x-embed, from the top of that markup where a rule put the element itself, or one holding it, in its own place, since where it stood before is no longer known, and the path of the element a rule was converting for a node the rule built itself, the markup itself up to 200 characters, a sentence for a person where one helps, and the class of the conversion rule that caused it.

Category	Meaning	Listed
carried	Markup the old page showed that no rule could convert was kept in a block an author rebuilds	once per event
dropped	Markup whose content could not survive was left out	once per event
degraded	Content was kept in a weaker form	once per event
unresolved	A reference to an entity or a file was kept as written, or a filter of the source format was not run	once per event; what concerns the source format, once per document
created	An entity the document names, such as a media item, is created when the document is written: would_create until the write has created it, created once it has; or an item a conversion created unpublished is published: would_publish, then published	once per event
refused	The checks a save runs refused the document	once per event
ignored	Something the document has no place for, which changes nothing a reader sees	counted, once per code and key
stripped	Markup removed on purpose, such as a script or an empty paragraph	counted, once per code and key

An entry's count is the number of events it stands for, and the totals drush blocknote:convert-html prints add the counts up. A counted category's entry counts every event of its code and key in the document. A listed category's entry counts one event, except that one narrowing of one block's text, such as a style the format refuses on several of its runs, is one entry counting each run.

The codes the module's own rules write:

Code	Category	What happened
block_not_permitted	degraded or dropped	The format does not enable the block; it became a paragraph, or was dropped, or, for a gallery a site's rule builds, each item became a media block
unknown_element	dropped	No rule converts the element, and it holds nothing, such as an embed placeholder
embed_not_converted	carried or dropped	Embedded content
media_url_refused, media_without_source	dropped	An image, video or audio element whose source the media allowlist refuses, or that names none
inline_content_not_permitted	dropped	Custom inline content the format does not enable
replacement_limit	dropped	A rule replaced an element more than eight times in a row, or more than 1000 elements were replaced in one document; one entry per limit counts every element it dropped
nesting_limit	dropped	An element nested more than 10000 deep, which a rule that keeps making the markup deeper reaches; one entry counts every element it dropped
block_in_inline_content	dropped	A block was met where a rule asked for inline content
content_not_carried	dropped	An element a rule or a class mapping made a block with no content, such as a divider, held text or media
prop_value_clamped, prop_value_not_permitted	degraded	A value the format does not permit, such as a heading level, became the nearest permitted one or the default
style_not_permitted, color_not_permitted, alignment_not_permitted, new_tab_not_permitted	degraded	Left off, as the format does not permit it
block_style_not_permitted	degraded	A block style a class mapping or a rule gave a block whose type the format lists no such style for, left off; keyed by the block type and the style's classes as a style line writes them, such as heading.lead
link_not_permitted, link_scheme_not_permitted, link_refused	degraded	The text kept without its link
code_style_dropped, subscript_dropped	degraded	Formatting the editor cannot show together
nested_link_dropped, image_link_dropped, code_block_link_dropped	degraded	A link inside a link or inside preformatted text, whose text was kept, or a link around a block
split_by_block	degraded	A block met inside text split it, such as a small image in a sentence
formatting_not_carried	degraded	Under breaks_as_paragraphs, line breaks inside more than 32 inline elements split a paragraph, and the text after them lost the formatting of the elements past the 32nd; one entry per document counts the splits
caption_as_paragraph	degraded	A table's caption, or a figure's that no single image takes, kept as a paragraph
details_without_summary	degraded	A disclosure without a summary, whose toggle has no text
layout_table_unwrapped	degraded	A table of one cell, whose content was kept in its place
table_too_large	degraded	A table whose grid, with those of the tables before it, has more slots than the empty cells the largest document holds, or that passes a table limit the site sets, whose content was kept in its place
table_cells_added, table_span_reduced	degraded	A table whose rows were filled with empty cells, or whose overlapping spans were made smaller; one entry per table counts the cells
table_cells_split, table_header_not_permitted	degraded	Merged cells or header cells the format does not permit; one entry per table counts them
table_header_not_carried	degraded	A th outside the leading rows and columns became an ordinary cell; one entry per table counts them
summary_flattened	degraded	A block in a disclosure's summary other than a paragraph or a heading, such as a list item, became a line of the toggle's text, keyed by its type; one entry per summary and key
table_cell_flattened	degraded	A block in a cell other than a paragraph became a line, keyed by its type, or the lines of a cell had different alignments or colors, keyed props; one entry per cell and key
nested_table_unwrapped, media_moved_after_table	degraded	A table inside a cell became lines, or a block a cell cannot hold was moved after the table
nesting_flattened	degraded	Blocks nested deeper than the site permits were moved up
list_renumbered	degraded	A numbered list right after another goes on with its numbers
heading_in_list_item, list_item_without_text	degraded	A list item, or a toggle whose summary is a heading the format cannot collapse, took the heading's text without its size, or a list item starting with another block has no text of its own
heading_id_not_carried	degraded	A heading's id that its page will not give it, because the heading's words make another one or the heading did not stay a heading, as in a list item, so a link to it with that fragment no longer reaches it; keyed by the old id. The comparison is made once the document is final, in the language of the entity the markup is read from, or the site's default language when no entity is named
anchor_not_carried	degraded	An <a>'s id or name, which a document cannot hold, so a link with that fragment no longer reaches its place; keyed by the id or the name, one entry for each. Not reported when the page gives a heading the same id, which the link then reaches
list_start_not_carried, preview_width_not_carried	degraded	A list's start number or a media element's width outside what a page shows, or a start number the list has no numbered item of its own to carry
list_attribute_not_carried	degraded or ignored	A list's type, which changed its markers, or a numbered list's reversed or its item's value, which changed its numbers, keyed by the attribute; ignored for reversed on a bulleted list and value on an item of one, which changed nothing a reader sees, and not reported for a type naming the markers the list has without one, or one a browser does not know
joined_single_paragraph, line_break_not_permitted	degraded	A single-paragraph format's join
invalid_utf8	degraded	Bytes that are not UTF-8, in the markup or in what a rule wrote, were replaced with U+FFFD
file_reference_kept_as_url	unresolved	An image, a video or an audio element naming a file by its uuid, or whose source is on the site, that no file was found for, kept as written; keyed file
private_file_kept_as_url	unresolved	An image, a video or an audio element, or a link, naming a file outside the public files directory that core's editor module did not record the entity the markup was read from as using, kept as written, since naming the file would serve it to every reader of the document; keyed file
entity_link_kept_as_url	unresolved	A link to content kept as a plain link: it named content by its attributes that was not found, or the format does not permit content links, or it has a query or a fragment, or it names a translation in another language than the document's; keyed by the entity type
link_alias_not_resolved	unresolved	A link through an alias, on a site that negotiates the language by domain, kept as a plain link, since the alias names content in the language of a host the address does not say; keyed domain
link_content_not_found, link_address_not_answered	unresolved	A link to an address of the site that names nothing, kept as a plain link: the canonical address of content the site no longer holds, keyed by the entity type, or an address no route, alias or file answers, keyed by its path once the language prefix is taken off and the alias resolved
image_style_not_carried	degraded	An image style's derivative, whose block shows the file it was made from; keyed by the style
substitution_not_permitted	degraded	A content link following an address rule the format does not permit, such as linkit's canonical, which follows the base rule; keyed by the rule
filter_not_applied, source_format_missing	unresolved	An enabled filter of the source format the converter does not run, keyed format.filter, or a source format that does not exist; once in every document's report, and once in a run's
remote_image_not_shown, local_image_not_shown	dropped	An image from elsewhere, or a source on the site that is no image file, which the source format's filter_html_image_secure replaced with an error icon
image_file_unchecked	unresolved	An image on the site kept under filter_html_image_secure though its file is not on the server running the conversion and no file was found for it, because it is under the public files directory or named as an image
site_host_unknown	unresolved	Images at an absolute address no --site-url names kept under filter_html_image_secure, because the conversion ran without the site's address; once in the report of every document holding one, and once in a run's
would_create, created	created	A media item the document names that no item was found for, or a file entity registered for a public file no file entity named, created when the document is written, as Media items and Files and content describe; keyed by the entity type and bundle, such as media.image or file.file
would_publish, published	created	A media item the running job created unpublished, which a page its readers see names, published when the document is written, as Media items describes; keyed by the entity type and bundle
media_not_found	unresolved	An embedded item the site does not hold, named by its uuid, or by an id, which leaves it out, keyed media; a player no item holds, kept as markup that could not be converted, keyed by the provider; or, under --images=media, an image that stays an image block for no reason reported otherwise, keyed image
media_not_created	unresolved	The conversion asked for an item and none is created, keyed by why: owner, private_file, type, type_not_permitted, type_fetches, address_too_long, address_not_provider, older_revision, uuid_taken, trashed or alt_missing
media_value_not_set	ignored	A site's rule gave an item it creates a value it may not set, or one its field does not validate, so the value was left off: only a field the site added to the media type that names no file or other entity takes a value; keyed by the field
media_type_not_permitted	unresolved	An item of a media type the format does not permit, keyed by the type
media_not_published	unresolved	An image or a player whose file or video only unpublished items hold, which stays an image block or is kept, keyed by the item's media type
embed_not_filtered, media_reference_invalid	dropped	An embedded item under a source format that does not run media_embed or entity_embed, or one naming no item by a uuid, or for entity_embed by a uuid or an id, or naming neither a display nor a view mode, which a page showed nothing for
view_mode_not_permitted, embed_display_not_carried	degraded	A view mode the format does not permit, or an entity_embed display that names none, shown in the default view mode; keyed by the view mode or the display
alt_shortened	degraded	Alternative text longer than the 512 characters a media block holds, cut there
video_start_not_carried, video_end_not_carried, playlist_not_carried, privacy_mode_not_carried	degraded	A player's start or end time or the playlist it played the video in, keyed by the provider, or YouTube's privacy-enhanced player, which the item does not carry
media_width_not_carried	ignored	An image's width, which a media block does not hold
caption_link_dropped	degraded	A link in a figure's caption, which holds plain text; one entry per caption counts them
callout_heading_as_title, callout_heading_as_paragraph	degraded	With BlockNote Extras, a heading a callout started with became its title, or another heading inside a callout became a paragraph, as Callouts describes; keyed heading
markup_not_repaired	degraded	Markup too large for its character references and control characters to be read as a browser reads them, which was parsed as stored
unsupported_blocks, and the validator's own codes such as too_many_blocks	refused	The checks a save runs refused the document
private_file_named	refused	A conversion rule built a document naming a file the converter would have kept as its address, as private_file_kept_as_url says; keyed by the file's identity
source_too_large, markup_too_deep, markup_too_many_attributes, conversion_too_slow	refused	The markup is larger than the conversion reads, nested so deep or carrying so many attributes on one element that parsing it would take too long, or its conversion took too long, as section 2 describes; the first is keyed by the markup's size in bytes, the third by the element's name
class_not_mapped	ignored	A class no class mapping names and no block style took, keyed tag.class
color_not_mapped, alignment_not_mapped, font_family	ignored	A value the document cannot carry
element_unwrapped	ignored	An element the converter does not know, whose content was kept
code_language_not_mapped, prop_not_declared, style_not_declared	ignored	An attribute or value with no place in the document
entity_attributes_not_filtered	ignored	An element's data-entity-type and data-entity-uuid that no filter of the source format read, editor_file_reference for a file, entity_links or linkit's for a link, so the page showed its address as written, which was read instead; keyed by the type they name
empty_block	stripped	A block holding nothing visible
not_displayed	stripped	Markup a page never shows as content
office_markup	stripped	An empty element a word processor wrote, such as <o:p>
control_character	stripped	A control character, keyed by its code point, such as U+0001
word_list_marker	stripped	The bullet or number Word wrote before the text of a list paragraph
class_stripped	stripped	An element a class mapping strips
new_tab_text_removed	stripped	The visually-hidden text inside a link with the blocknote-link--new-tab class, which the new-tab flag says again; keyed tag.visually-hidden

A conversion rule writes codes of its own.

8. Writing conversion rules

A site's markup carries meaning the module cannot know: a custom element an editor plugin stored, a class that marks a callout, a token an older filter expanded. A module adds conversion rules for those, as docs/extending.md section 14 describes, and drush blocknote:convert-html shows what they make of a site's own markup.

9. Converting a field's values

drush blocknote:convert <entity_type> <source_field> <target_field> converts the values of a text, text_long or text_with_summary field into a BlockNote document field of the same entity type. The document field is a new field beside the old one; the old one is never changed, and deleting it is the last step, taken by hand once the conversion is done and checked.

The procedure

1. Add the document field to each bundle, with the BlockNote format the documents are to be written in among its allowed formats, translatable when the text field is, and holding at least as many values. Leave the text field where it is. With the media submodule, permit in that format the view modes the old pages showed embedded items in: those the source format's Embed media filter allows, and its Default view mode, which an item naming none was shown in. An item in a view mode the format does not permit is shown in the default view mode instead, as Media items says.
2. Rehearse. Run the command with --dry-run, which converts and reports and writes nothing, and read the reports. Write conversion rules for the site's own markup (docs/extending.md section 14) and try them on single values with drush blocknote:convert-html. A dry run keeps a place of its own, so pass --restart to rehearse again from the start. It converts as the run does and saves nothing, and the current pass took about three times as long as the dry run in the module's own rehearsal, which helps size the maintenance window. With the media submodule, decide what images and players become and which media types new items go in, as Media items describes, and rehearse with the same --images, --create-media, --media-type and --media-owner the run will have: the dry run reports each item it would create as would_create. Before a run that creates remote video items, tick Queue thumbnail downloads on their media type, or, for a type that fetches, create the items first with drush blocknote:create-videos, as Creating video items first describes.
3. Convert the current revisions in a maintenance window, since an author saving an entity while it is converted makes the run pass over it: drush blocknote:convert node body field_body. Run it with Drush's --uri set to the site's address, as The source format explains, with --site-url naming the addresses the site was served at before and --legacy-file-path the paths its files were, as Files and content explains. The files the documents name need their file entities on the site, migrated with the content, or every image is kept as its address and reported. Modules that react to a save see each revision the run writes saved: pathauto generates again the alias of each default revision it makes aliases for, rewriting one its pattern now gives otherwise unless its update action is Do nothing, and Search API marks each entity the pass saves changed in every index tracking it, and registers it to be indexed once the request ends where the index indexes items immediately, so set both for the window and index afterwards. The older pass (step 6) sets search_api_skip_tracking on its saves, which Search API reads as a request not to track them, so only the current pass marks anything. Once the pass is done, clear the entity cache with drush cache:clear bin entity: the run leaves in it roughly a row for each revision it wrote, which the save made stale, and one for each entity it read, fewer where a save cleared it, since saving a revision deletes its entity's own row.
4. Switch the form and the display to the document field in the same window, before authors are let back in: its widget in the form in place of the text field's, and its formatter in each display in place of the text field's. A display that shows the text through core's Trimmed formatter, as a teaser or a card often does, switches to this module's Trimmed formatter, with its Limit at the core formatter's Trimmed limit, counting characters and cutting after whole blocks only, the nearest to core's preference for the end of a paragraph, and with Text where it cuts emptied, since core appends nothing. Core counted the characters of the filtered markup, tags included, where this formatter counts the text a visitor reads, so the same limit shows more text, and a lower one keeps a teaser the length it was. A display that uses core's Summary or trimmed formatter switches the same way, and when the text field held summaries, its Summary field names the document field --summary-target converted them into, so that a summary is shown whole wherever there is one, as core showed it. docs/rendering.md section 12 says what a trimmed rendering shows. The theme's styles for the text field do not all carry over: a rule scoped to the class a text field's template gives its wrapper, such as .text-formatted, which Claro and core's Starterkit theme add, does not reach the document field, whose markup docs/rendering.md describes, and an image block aligned to the left or the right no longer floats with the text wrapped around it, as its section 8 says, while a media item aligned to the left or the right floats as core floated it, as long as the formatter's Attach the default styles is on. Re-point every Search API index field that reads the text field at the document field, as the README's Search indexing says, before the text field is deleted. An author who edits the text field once its value is converted changes text the document field does not show, and the change is lost when the text field is deleted. The closing line of every run that finishes and the status report count such revisions and name their entities, and --overwrite with --ids converts them again.
5. Check the reports, the documents that hold markup that could not be converted, which the status report counts, and the pages.
6. Convert the older revisions with --revisions=older, or decide not to: a revision left unconverted shows an empty document field once it is reverted to, and its text is gone once the text field is deleted. The older pass may run while the site is open, at a quiet time: saving each revision it writes invalidates its entity's cache tags and its entity type's list tag, such as node_list, so every page that lists content of that type is built again after each revision written. It reads every entity of the bundles to tell their current revisions apart, so clear the entity cache again once it is done. Search API is not asked to index again for these saves, as step 3 says, whether or not it is installed; the default revision's save stays tracked.
7. Delete the text field, once nothing shows it. Run the command once more first: a conversion that is done reads nothing, and counts again the revisions left and the text changed after it was converted.

What a run does

The current pass reads each entity and converts its current revisions: the default revision, and the revision an editor of each translation sees, which is content moderation's draft where there is one. Every translation of such a revision is converted. The older pass, --revisions=older, converts every other revision, in the order of its id; --revisions=all runs both, the current revisions first. An entity type that keeps no revisions has only the entity, which the current pass converts: --revisions=older is refused as a usage error, with exit code 2, and --revisions=all runs the current pass alone.

An entity in the trash of the Trash module is converted as any other, and so is the revision that moved an entity there, which stays among its older revisions once the entity is restored: Trash hides both from entity queries and loads, and the passes and the closing counts read them with the trash ignored, so that an entity restored once the text field is deleted shows its document. Writing a trashed entity leaves it in the trash. Its values are still converted as the site's pages read the site, so content, a file or a media item in the trash is not found, as a page did not find it. A revision is saved, and its documents announced, as the run's caller would save it, so an entity a module deletes as the save runs goes to the trash; only a trashed revision, or a revision of a trashed entity, is saved and announced with the trash ignored, since core reads it again as it saves, and an entity a module deletes then is deleted for good, as Trash deletes it outside its active context.

Each revision is written where it is. No revision is added, the default revision stays the default, content moderation's draft stays a draft, and no changed time moves; only the document field is written. A revision is read again just before it is written, under a lock on its entity's row, and is passed over, as skipped_changed_during_run, when anything it was converted from or into changed since, or, in the current pass, when a newer revision was saved in the meantime. A host whose bundle was moderated after it was created gains a moderation state for each revision written, with the publishing status it had.

A document field that already holds a document is left alone. When it holds what the conversion writes, nothing is saved; when it holds anything else, the value is reported as skipped_target_differs. --overwrite replaces a document the converter wrote and nobody has changed since, such as one converted before a rule was improved or from text an author changed afterwards, and never one an author wrote; a run without it says, for a document the converter wrote, that --overwrite replaces it. A field is written whole: when one of its values in a translation is refused, none of them is written, so that once the value converts the field is written in one piece. A value that is empty, or converts into no block, has no document, so in a field of several values the documents after it move up a position.

Each document's identity is derived from the entity, the language, the field and the position of the value it was converted from, which is where it is stored unless a value before it has no document, so it is the same in every revision and in every run, and converting the same value again writes the same bytes. Each document written into a current revision is announced to modules observing documents with the change origin converter, and the older revisions are written without an announcement, as docs/workflows.md section 10 describes.

The summary of a text_with_summary field is left out, and reported as summary_dropped with its value, unless --summary-target names a document field to convert it into, in the format --summary-format names or the one that field allows.

Resuming, limits and options

A run keeps its place, and running the command again goes on from where the last run stopped, whether it stopped at --limit, out of memory or by a defect; a run that has finished the passes it was asked for reads nothing more. The place moves past each entity, or each revision in the older pass, only once its rows are written to the report and its outcomes added to the totals, and the length of each report file is stored with it; a run that resumes first cuts the files back to that length, so a run stopped anywhere, even between writing an entity's rows and moving the place past them, reports each value once. An entity or revision whose conversion or write throws is reported as failed, with what was thrown, and the run goes on past it; once the cause is fixed, --ids converts it. Ten entities in a row stop the run before the first of them, since that is more likely the database or a rule failing for every value, and the next run reads them again; in the older pass the revisions of one entity count as one, and --max-failures sets how many, 0 for a run that never stops. --restart starts over, and with --overwrite converts every entity again, writing only where the conversion differs from what the converter wrote before. A run that would resume a conversion started with another format, other bundles, another summary field, other options, --unconverted, --breaks-as-paragraphs, --rule-option, --site-url and --legacy-file-path, or at another address, Drush's --uri, whose host and base path say what an address names, is refused; --site-url and --legacy-file-path are compared as they are read, so example.com and https://Example.com:443/ are one address; --max-source-bytes and --max-seconds may change between runs. Two runs of the same conversion never run at once: the second stops with exit code 4. A run renews its lock before it converts each entity or revision and again before it writes its rows and moves the place, and one whose lock expired and was taken by another run stops with exit code 4 too, recording nothing more; a revision it wrote in the meantime is found written by the run that took the lock. --ids converts the entities it names, keeping no place. --bundle limits a run to some bundles, every bundle holding both fields by default, and --batch-size sets how many entities or revisions are read at a time, $settings['entity_update_batch_size'] by default.

--unconverted, --breaks-as-paragraphs, --register-files, --rule-option, --site-url, --legacy-file-path, --max-source-bytes and --max-seconds are the options of section 2, applied to every value.

The report

Each run writes two files into --report-dir, by default a directory in temporary:// itself named after the conversion, the time it started in UTC and random characters, which nobody else can make first, which a resumed run goes on writing in. Since the reports quote the content they convert, a directory where the web server serves files is refused: anywhere under the site's web root, which is where Drush reads a relative path from, or in the directory of public://, assets:// or another stream wrapper of their kind, however the directory is spelled, a scheme in capitals, a path through .., a symbolic link or letters in another case included. The directory is a local path, a file:// URI or a URI of a local stream wrapper Drupal registers, such as temporary:// or private://; any other stream PHP can open, such as ftp:// or compress.zlib://, is refused. Each directory the command makes for the reports, and each file it writes, can be read by its owner alone. A directory, or a report file in it, that another user owns or that others than its owner can write is refused, and a report file that is a symbolic link is not written to. A cell a spreadsheet would read as a formula, such as one starting with =, is written after an apostrophe.

documents.csv has a row for each value: entity_type, entity_id, bundle, revision_id, current (1 when the revision is the default one or the one an editor of the translation sees), langcode, field (the document field written), delta, source_format, target_format, outcome, blocks (nested ones included), bytes, source_hash, a count for each category of section 7, and detail, which says why a value was passed over or names the codes of its refusal.

Outcome	Meaning
written	The document was written
would_write	A dry run would have written it
unchanged	The field already held this document
skipped_target_differs	The field holds another document, which was left alone
skipped_empty_source	The value was empty
empty_result	The value converted into no block, such as <p>&nbsp;</p>, and nothing was written for it
refused	The checks a save runs refused the document, or it was too large or too slow to convert
skipped_with_refused	Another value of the same field and translation was refused
skipped_changed_during_run, skipped_revision_gone	The revision changed, or was deleted, while the run converted it
failed	Converting or writing the entity's current revisions, or the older revision, threw; detail names what was thrown and where, the row names no language or value, and anything written before it threw stays written

elements.csv has a row for each report entry of a value in a current revision, with the columns of section 7 after the value's entity_type, entity_id, revision_id, langcode, field and delta. An older revision's entries mostly repeat those of the revision after it, so only its refusals are listed; documents.csv counts the rest.

The command prints how many values had each outcome in this run and in every run of the conversion, and each code and key with its events and the number of documents it appears in, the codes every value of a source format carries counted once for each key. A run that finishes the passes it was asked for ends saying how many revisions hold text and no document, when there are any: older revisions the older pass has not converted, and revisions whose values were refused or converted into nothing, which stay counted while the text field holds them, since no pass writes them; for an entity type that keeps no revisions, it counts the entities whose values were refused or converted into nothing. It also says how many current revisions hold text an author changed after it was converted, read from the entities saved since the conversion started where the entity type records a changed time, and names their entities for --overwrite --ids. The status report says the same, as of the last run that finished, until the text field is deleted.

Exit code	Meaning
0	The run did what it was asked, and nothing was refused
1	A value was refused; documents.csv names it
2	The command was called wrongly, or the site does not hold what the conversion needs, such as a field of the wrong type or a format the field does not allow; nothing was read
3	Something threw once the run started, which is a defect in the converter, in a conversion rule or in what a save runs: an entity or revision was reported as failed and the run went on past it, or the run stopped; what was written stays written, and running the command again goes on after the last entity or revision reported
4	Another run of the same conversion is going on, or took the lock of this one once it expired

Converting one value from code

A module converting values one at a time, such as a migration's process plugin, calls blocknote.html_converter with the value, the format, a ConversionSource naming the source format and, where there is one, the entity, and its options, among them ConversionOptions::SITE_URLS and LEGACY_FILE_PATHS, lists of strings, for which the converter throws an \InvalidArgumentException when an entry is no site address or path. It gets what the command gets: the guard, the block for markup that could not be converted and every report entry, a source format's included. A value without an entity has its block ids derived from its markup, and its document is given an identity by the field when it is saved. The module ships no migrate process plugin yet.