Class: MaquinaStream::Renderer::TagBlocks

Inherits:
Object
  • Object
show all
Defined in:
lib/maquina_stream/renderer/tag_blocks.rb

Overview

Gives a registered app-meaning tag its own HTML block, by putting blank lines around its opening and closing tags before commonmarker ever sees the buffer.

MaquinaStream::Renderer::TagBlocks.call(markdown, names: MaquinaStream.tags.keys)

The bug this exists for

CommonMark ends an HTML block at the first blank line (spec §4.6, condition 7). A tag a model wrote across several paragraphs therefore has its closing tag emitted inside a paragraph:

<thinking>\nFirst para.\n\nSecond para.\n</thinking>\n\nAfter the tag.
#  <thinking>
#  First para.
#  <p>Second para.<br />\n</thinking></p>
#  <p>After the tag.</p>

The HTML5 parser sees a stray end tag, ignores it, and never closes <thinking> — so the rest of the message is parsed inside it and the host's partial is handed content the model wrote after the tag closed. That is a content leak, not a cosmetic defect: the partial is where the host says "this is the model's private reasoning".

With a blank line after the opener and before the closer, each tag is its own HTML block, the element closes where the model closed it, and the paragraphs between them are ordinary markdown.

Why it normalises markdown rather than HTML

Rebalancing end tags in the commonmarker output would mean hand-editing an HTML string that is model output — the one thing this engine never does. This pass inserts newlines into the markdown and changes nothing else; the trust boundary stays exactly where it was, with the sanitizer running last over parsed HTML.

What it will not touch

It is a privilege boundary, and it is written to be boring:

  • Only registered names. names: is what the host passed to MaquinaStream.register_tag. An unregistered <script>, <iframe> or <img> is not matched, not moved and not re-parsed. With an empty registry the input is returned byte for byte.
  • Code wins. Matching runs over MaquinaRemend::Scanner#masked_text, which blanks fenced code blocks and balanced inline code spans while preserving offsets, so a <thinking> inside a fence or a backtick span is text and stays text. Indented code is covered by the indent rule below rather than by the scanner, which does not track it, and the raw HTML regions the scanner does not model either — comments, <script>, <pre> and friends — are masked here by RAW_REGIONS.
  • Complete tags only. A match is a whole open or close tag on one line, with CommonMark's attribute grammar — quoted values may contain >, and a tag broken across a newline is not a tag. <thinkingXYZ> does not match thinking; <thinking/> opens nothing.
  • Block position only. The opening tag must begin its line under four columns of indent, which is the shape that starts an HTML block and therefore the shape that leaks. Anything deeper may be indented code or list content and is left alone; an inline <citation>…</citation> in the middle of a sentence is left alone because it already works, and is what the registry was built for.
  • Pairs only. An opener with no closer, or a closer with no opener, inserts nothing. Nesting is matched innermost-first, as HTML does it.

Insertion is idempotent: a buffer that already has the blank lines comes back byte-identical, which is what makes the pass safe to run on every frame of a stream.

Constant Summary collapse

MAX_INDENT =

An HTML block opener may be indented up to three columns; the fourth makes it indented code.

3
BLOCK_INDENT =

An indent that still starts an HTML block, and one that is deep enough to be indented code or list content instead.

/\A {0,#{MAX_INDENT}}\z/
CODE_INDENT = 
/\A[ \t]{#{MAX_INDENT + 1},}\z/
ATTRIBUTE =

CommonMark's attribute grammar (spec §6.6), restricted to spaces and tabs. A tag whose attributes wrap onto a second line is not a complete tag for HTML-block purposes, so it must not be one for us either.

/[ \t]+[a-zA-Z_:][a-zA-Z0-9_.:-]*(?:[ \t]*=[ \t]*(?:[^ \t"'=<>`]+|'[^']*'|"[^"]*"))?/
RAW_REGIONS =

The raw regions CommonMark reads to a closing marker rather than to a blank line: HTML block types 1 through 5 — <script>, <pre>, <style> and <textarea>, comments, processing instructions, declarations and CDATA.

MaquinaRemend::Scanner masks fences and inline code, which is the markdown half of "this is text, not markup"; it does not model these, because no repair it makes has ever needed to. They matter here for one reason: a blank line inserted inside one of them ends it early, and text the model had buried in a comment or a <script> — text the sanitizer would have dropped whole — comes back out as live markdown. Masking them means a registered name written inside one is never a tag.

An unterminated region masks to the end of the buffer, which is the conservative answer while a message is still streaming.

[
  /<!--.*?(?:-->|\z)/m,
  /<!\[CDATA\[.*?(?:\]\]>|\z)/m,
  /<\?.*?(?:\?>|\z)/m,
  /<![A-Za-z].*?(?:>|\z)/m,
  %r{<(script|pre|style|textarea)\b.*?(?:</\1\s*>|\z)}mi
].freeze

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(names:) ⇒ TagBlocks

Returns a new instance of TagBlocks.



127
128
129
 
# File 'lib/maquina_stream/renderer/tag_blocks.rb', line 127

def initialize(names:)
  @names = Array(names).map { |name| name.to_s.downcase }.reject(&:empty?).uniq
end

Instance Attribute Details

#names ⇒ Object (readonly)

The registered tag names this instance will normalise, downcased.



125
126
127
 
# File 'lib/maquina_stream/renderer/tag_blocks.rb', line 125

def names
  @names
end

Class Method Details

.call(markdown, names: MaquinaStream.tags.keys) ⇒ Object

Normalises markdown for the given tag names and returns it. Returns the argument unchanged when nothing is registered, when no registered tag appears in block position, or when the blank lines are already there.



119
120
121
 
# File 'lib/maquina_stream/renderer/tag_blocks.rb', line 119

def call(markdown, names: MaquinaStream.tags.keys)
  new(names: names).call(markdown)
end

Instance Method Details

#call(markdown) ⇒ Object 



131
132
133
134
135
136
137
138
 
# File 'lib/maquina_stream/renderer/tag_blocks.rb', line 131

def call(markdown)
  return markdown if markdown.nil? || markdown.empty? || names.empty?

  insertions = plan(markdown.to_s)
  return markdown if insertions.empty?

  splice(markdown.to_s, insertions)
end