Module:Wikitext Parsing
.svg/60px-Test_Template_Info-Icon_-_Version_(2).svg.png){kind=icon}
 | This Lua module is used on approximately 17,700,000 pages, or roughly 28% of all pages. To avoid major disruption and server load, any changes should be tested in the module's /sandbox or /testcases subpages, or in your own module sandbox. The tested changes can be added to this page in a single edit. Consider discussing changes on the talk page before implementing them. |
 | This module can only be edited by administrators because it is transcluded onto one or more cascade-protected pages. |
This module provides some functions to help with the potential complex situation involved in modules like Module:Template parameter value, which intend to process the raw wikitext of a page and want to respect nowiki tags or similar reliably. This module is designed only to be called by other modules.
PrepareText
 | This module is rated as ready for general use. It has reached a mature form and is thought to be relatively bug-free and ready for use wherever appropriate. It is ready to mention on help pages and other Wikipedia resources as an option for new users to learn. To reduce server load and bad output, it should be improved by sandbox testing rather than repeated trial-and-error editing. |
 | This module is subject to page protection. It is a highly visible module in use by a very large number of pages, or is substituted very frequently. Because vandalism or mistakes would affect many pages, and even trivial editing might cause substantial load on the servers, it is protected from editing. |
PrepareText(text, keepComments) will run any content within certain tags that disable processing (<nowiki>, <pre>, <syntaxhighlight>, <source>, <math>) through mw.text.nowiki and remove HTML comments to avoid irrelevant text being processed by modules, allowing tricky syntax to be parsed through more basic means such as %b{}.
If the second parameter, keepComments, is set to true, the content of HTML comments will be passed through mw.text.nowiki instead of being removed entirely.
Any code using this function directly should consider using mw.text.decode to correct the output at the end if part of the processed text is returned, though this will also decode any input that was encoded but not inside a no-processing tag, which likely isn't a significant issue but still something worth considering.
ParseTemplates
 | This module is rated as beta, and is ready for widespread use. It is still new and should be used with some caution to ensure the results are as expected. |
| This module is subject to page protection. It is a highly visible module in use by a very large number of pages, or is substituted very frequently. Because vandalism or mistakes would affect many pages, and even trivial editing might cause substantial load on the servers, it is protected from editing. |
ParseTemplates(InputText, dontEscape) will attempt to parse all {{Templates}} on a page, handling multiple factors such as [[Wikilinks]] and {{{Variables}}} among other complex syntax. Due to the complexity of the function, it is considerably slow, and should be used carefully. The function returns a list of template objects in order of appearance, which have the following properties:
- Args: A key-value set of arguments, not in order
- ArgOrder: A list of keys in the order they appear in the template
- Children: A list of template objects that are contained within the existing template, in order of appearance. Only immediate children are listed
- Name: The name of the template
- Text: The raw text of the template
If the second parameter, dontEscape, is set to true, the inputted text won't be ran through the PrepareText function.
require("strict")--Helper functionslocalfunctionstartswith(text,subtext)returnstring.sub(text,1,#subtext)==subtextendlocalfunctionendswith(text,subtext)returnstring.sub(text,-#subtext,-1)==subtextendlocalfunctionallcases(s)returns:gsub("%a",function(c)return"["..c:upper()..c:lower().."]"end)endlocaltrimcache={}localwhitespace={[" "]=1,["\n"]=1,["\t"]=1,["\r"]=1}localfunctioncheaptrim(str)--mw.text.trim is surprisingly expensive, so here's an alternative approachlocalquick=trimcache[str]ifquickthenreturnquickelse-- local out = string.gsub(str, "^%s*(.-)%s*$", "%1")locallowEndfori=1,#strdoifnotwhitespace[string.sub(str,i,i)]thenlowEnd=ibreakendendifnotlowEndthentrimcache[str]=""return""endfori=#str,1,-1doifnotwhitespace[string.sub(str,i,i)]thenlocalout=string.sub(str,lowEnd,i)trimcache[str]=outreturnoutendendendend--[=[ Implementation notes---- NORMAL HTML TAGS ----Tags are very strict on how they want to start, but loose on how they end.The start must strictly follow <[tAgNaMe](%s|>) with no room for whitespace inthe tag's name, but may then flow as they want afterwards, making<div\nclass\n=\n"\nerror\n"\n> validThere's no sense of escaping < or >E.g. <div class="error\>"> will end at \> despite it being inside a quote <div class="<span class="error">error</span>"> will not process the larger divIf a tag has no end, it will consume all text instead of not processing---- NOPROCESSING TAGS (nowiki, pre, syntaxhighlight, source, etc.) ----(In most comments, <source> will not be mentioned. This is because it is thedeprecated version of <syntaxhighlight>)No-Processing tags have some interesting differences to the above rules.For example, their syntax is a lot stricter. While an opening tag appears tofollow the same set of rules, A closing tag can't have any sort of extraformatting period. While </div a/a> is valid, </nowiki a/a> isn't - onlynewlines and spaces/tabs are allowed in closing tags.Note that, even though <pre> tags cause a visual change when the ending tag hasextra formatting, it won't cause the no-processing effects. For some reason, theformat must be strict for that to apply.Both the content inside the tag pair and the content inside each side of thepair is not processed. E.g. <nowiki |}}>|}}</nowiki> would have both of the |}}escaped in practice.When something in the code is referenced to as a "Nowiki Tag", it means a tagwhich causes wiki text to not be processed, which includes <nowiki>, <pre>,and <syntaxhighlight>Since we only care about these tags, we can ignore the idea of an interceptingtag preventing processing, and just go straight for the first ending we can findIf there is no ending to find, the tag will NOT consume the rest of the text interms of processing behaviour (though <pre> will appear to have an effect).Even if there is no end of the tag, the content inside the opening half willstill be unprocessed, meaning {{X20|<nowiki }}>}} wouldn't end at the first }}despite there being no ending to the tag.Note that there are some tags, like <math>, which also function like <nowiki>which are included in this aswell. Some other tags, like <ref>, have far toounpredictable behaviour to be handled currently (they'd have to be split andprocessed as something seperate - its complicated, but maybe not impossible.)I suspect that every tag listed in [[Special:Version]] may behave somewhat likethis, but that's far too many cases worth checking for rarely used tags that maynot even have a good reason to contain {{ or }} anyways, so we leave them alone.---- HTML COMMENTS AND INCLUDEONLY ----HTML Comments are about as basic as it could get for thisStart at <!--, end at -->, no extra conditions. Simple enoughIf a comment has no end, it will eat all text instead of not being processedincludeonly tags function mostly like a regular nowiki tag, with the exceptionthat the tag will actually consume all future text if not given an ending asopposed to simply giving up and not changing anything. Due to complications andthe fact that this is far less likely to be present on a page, aswell as beingsomething that may not want to be escaped, includeonly tags are ignored duringour processing--]=]localvalidtags={nowiki=1,pre=1,syntaxhighlight=1,source=1,math=1}--This function expects the string to start with the taglocalfunctionTestForNowikiTag(text,scanPosition)localtagName=(string.match(text,"^<([^\n />]+)",scanPosition)or""):lower()ifnotvalidtags[tagName]thenreturnnilendlocalnextOpener=string.find(text,"<",scanPosition+1)or-1localnextCloser=string.find(text,">",scanPosition+1)or-1ifnextCloser>-1and(nextOpener==-1ornextCloser<nextOpener)thenlocalstartingTag=string.sub(text,scanPosition,nextCloser)--We have our starting tag (E.g. '<pre style="color:red">')--Now find our ending...ifendswith(startingTag,"/>")then--self-closing tag (we are our own ending)return{Tag=tagName,Start=startingTag,Content="",End="",Length=#startingTag}elselocalendingTagStart,endingTagEnd=string.find(text,"</"..allcases(tagName).."[ \t\n]*>",scanPosition)ifendingTagStartthen--Regular tag formationlocalendingTag=string.sub(text,endingTagStart,endingTagEnd)localtagContent=string.sub(text,nextCloser+1,endingTagStart-1)return{Tag=tagName,Start=startingTag,Content=tagContent,End=endingTag,Length=#startingTag+#tagContent+#endingTag}else--Content inside still needs escaping (also linter error!)return{Tag=tagName,Start=startingTag,Content="",End="",Length=#startingTag}endendendreturnnilendlocalfunctionTestForComment(text,scanPosition)--Like TestForNowikiTag but for <!-- -->ifstring.match(text,"^<!%-%-",scanPosition)thenlocalcommentEnd=string.find(text,"-->",scanPosition+4,true)ifcommentEndthenreturn{Start="<!--",End="-->",Content=string.sub(text,scanPosition+4,commentEnd-1),Length=commentEnd-scanPosition+3}else--Consumes all text if not given an endingreturn{Start="<!--",End="",Content=string.sub(text,scanPosition+4),Length=#text-scanPosition+1}endendreturnnilend--[[ Implementation notesThe goal of this function is to escape all text that wouldn't be parsed if itwas preprocessed (see above implementation notes).Using keepComments will keep all HTML comments instead of removing them. Theywill still be escaped regardless to avoid processing errors--]]localfunctionPrepareText(text,keepComments)localnewtext={}localscanPosition=1whiletruedolocalNextCheck=string.find(text,"<[NnSsPpMm!]",scanPosition)--Advance to the next potential tag we care aboutifnotNextCheckthen--Donenewtext[#newtext+1]=string.sub(text,scanPosition)breakendnewtext[#newtext+1]=string.sub(text,scanPosition,NextCheck-1)scanPosition=NextChecklocalComment=TestForComment(text,scanPosition)ifCommentthenifkeepCommentsthennewtext[#newtext+1]=Comment.Start..mw.text.nowiki(Comment.Content)..Comment.EndendscanPosition=scanPosition+Comment.LengthelselocalTag=TestForNowikiTag(text,scanPosition)ifTagthenlocalnewTagStart="<"..mw.text.nowiki(string.sub(Tag.Start,2,-2))..">"localnewTagEnd=Tag.End==""and""or--Respect no tag ending"</"..mw.text.nowiki(string.sub(Tag.End,3,-2))..">"localnewContent=mw.text.nowiki(Tag.Content)newtext[#newtext+1]=newTagStart..newContent..newTagEndscanPosition=scanPosition+Tag.Lengthelse--Nothing special, move on...newtext[#newtext+1]=string.sub(text,scanPosition,scanPosition)scanPosition=scanPosition+1endendendreturntable.concat(newtext,"")end--[=[ Implementation notesThis function is an alternative to Transcluder's getParameters which considersthe potential for a singular { or } or other odd syntax that %b doesn't like tobe in a parameter's value.When handling the difference between {{ and {{{, mediawiki will attempt to matchas many sequences of {{{ as possible before matching a {{E.g. {{{{A}}}} -> { {{{A}}} } {{{{{{{{Text|A}}}}}}}} -> {{ {{{ {{{Text|A}}} }}} }}If there aren't enough triple braces on both sides, the parser will compromisefor a template interpretation.E.g. {{{{A}} }} -> {{ {{ A }} }}While there are technically concerns about things such as wikilinks breakingtemplate processing (E.g. {{[[}}]]}} doesn't stop at the first }}), it shouldn'tbe our job to process inputs perfectly when the input has garbage ({ / } isn'tlegal in titles anyways, so if something's unmatched in a wikilink, it'sguaranteed GIGO)Setting dontEscape will prevent running the input text through EET. Avoidsetting this to true if you don't have to set it.Returned values:A table of all templates. Template data goes as follows: Text: The raw text of the template Name: The name of the template Args: A list of arguments Children: A list of immediate template children--]=]--Helper functionslocalfunctionboundlen(pair)returnpair.End-pair.Start+1end--Main functionlocalfunctionParseTemplates(InputText,dontEscape)--SetupifnotdontEscapethenInputText=PrepareText(InputText)endlocalfunctionfinalise(text)ifnotdontEscapethenreturnmw.text.decode(text)elsereturntextendendlocalfunctionCreateContainerObj(Container)Container.Text={}Container.Args={}Container.ArgOrder={}Container.Children={}-- Container.Name = nil-- Container.Value = nil-- Container.Key = nilContainer.BeyondStart=falseContainer.LastIndex=1Container.finalise=finalisefunctionContainer:HandleArgInput(character,internalcall)ifnotinternalcallthenself.Text[#self.Text+1]=characterendifcharacter=="="thenifself.Keythenself.Value[#self.Value+1]=characterelseself.Key=cheaptrim(self.Valueandtable.concat(self.Value,"")or"")self.Value={}endelse--"|" or "}"ifnotself.Namethenself.Name=cheaptrim(self.Valueandtable.concat(self.Value,"")or"")self.Value=nilelseself.Value=self.finalise(self.Valueandtable.concat(self.Value,"")or"")ifself.Keythenself.Key=self.finalise(self.Key)self.Args[self.Key]=cheaptrim(self.Value)self.ArgOrder[#self.ArgOrder+1]=self.KeyelselocalKey=tostring(self.LastIndex)self.Args[Key]=self.Valueself.ArgOrder[#self.ArgOrder+1]=Keyself.LastIndex=self.LastIndex+1endself.Key=nilself.Value=nilendendendfunctionContainer:AppendText(text,ftext)self.Text[#self.Text+1]=(ftextortext)ifnotself.Valuethenself.Value={}endself.BeyondStart=self.BeyondStartor(#table.concat(self.Text,"")>2)ifself.BeyondStartthenself.Value[#self.Value+1]=textendendfunctionContainer:Clean(IsTemplate)self.Text=table.concat(self.Text,"")ifself.ValueandIsTemplatethenself.Value={string.sub(table.concat(self.Value,""),1,-3)}--Trim ending }}self:HandleArgInput("|",true)--Simulate endingendself.Value=nilself.Key=nilself.BeyondStart=nilself.LastIndex=nilself.finalise=nilself.HandleArgInput=nilself.AppendText=nilself.Clean=nilendreturnContainerend--Step 1: Find and escape the content of all wikilinks on the page, which are stronger than templates (see implementation notes)localscannerPosition=1localwikilinks={}localopenWikilinks={}whiletruedolocalPosition,_,Character=string.find(InputText,"([%[%]])%1",scannerPosition)ifnotPositionthen--DonebreakendscannerPosition=Position+2--+2 to pass the [[ / ]]ifCharacter=="["then--Add a [[ to the pending wikilink queueopenWikilinks[#openWikilinks+1]=Positionelse--Pair up the ]] to any available [[if#openWikilinks>=1thenlocalstart=table.remove(openWikilinks)--Pop the latest [[wikilinks[start]={Start=start,End=Position+1,Type="Wikilink"}--Note the pairendendend--Step 2: Find the bounds of every valid template and variable ({{ and {{{)localscannerPosition=1localtemplates={}localvariables={}localopenBrackets={}whiletruedolocalStart,_,Character=string.find(InputText,"([{}])%1",scannerPosition)ifnotStartthen--Done (both 9e9)breakendlocal_,End=string.find(InputText,"^"..Character.."+",Start)scannerPosition=Start--Get to the {{ / }} setifCharacter=="{"then--Add the {{+ set to the queueopenBrackets[#openBrackets+1]={Start=Start,End=End}else--Pair up the }} to any available {{, accounting for {{{ / }}}localBracketCount=End-Start+1whileBracketCount>=2and#openBrackets>=1dolocalOpenSet=table.remove(openBrackets)ifboundlen(OpenSet)>=3andBracketCount>=3then--We have a {{{variable}}} (both sides have 3 spare)variables[OpenSet.End-2]={Start=OpenSet.End-2,End=scannerPosition+2,Type="Variable"}--Done like this to ensure chronological orderBracketCount=BracketCount-3OpenSet.End=OpenSet.End-3scannerPosition=scannerPosition+3else--We have a {{template}} (both sides have 2 spare, but at least one side doesn't have 3 spare)templates[OpenSet.End-1]={Start=OpenSet.End-1,End=scannerPosition+1,Type="Template"}--Done like this to ensure chronological orderBracketCount=BracketCount-2OpenSet.End=OpenSet.End-2scannerPosition=scannerPosition+2endifboundlen(OpenSet)>=2then--Still has enough data left, leave it inopenBrackets[#openBrackets+1]=OpenSetendendendscannerPosition=End--Now move past the bracket setend--Step 3: Re-trace every object using their known bounds, collecting our parameters with (slight) easelocalscannerPosition=1localactiveObjects={}localfinalObjects={}whiletruedolocalLatestObject=activeObjects[#activeObjects]--Commonly needed objectlocalNNC,_,Character--NNC = NextNotableCharacterifLatestObjectthenNNC,_,Character=string.find(InputText,"([{}%[%]|=])",scannerPosition)elseNNC,_,Character=string.find(InputText,"([{}])",scannerPosition)--We are only after templates right nowendifnotNNCthenbreakendifNNC>scannerPositionandLatestObjectthenlocalscannedContent=string.sub(InputText,scannerPosition,NNC-1)LatestObject:AppendText(scannedContent,finalise(scannedContent))endscannerPosition=NNC+1ifCharacter=="{"orCharacter=="["thenlocalContainer=templates[NNC]orvariables[NNC]orwikilinks[NNC]ifContainerthenCreateContainerObj(Container)ifContainer.Type=="Template"thenContainer:AppendText("{{")scannerPosition=NNC+2elseifContainer.Type=="Variable"thenContainer:AppendText("{{{")scannerPosition=NNC+3else--WikilinkContainer:AppendText("[[")scannerPosition=NNC+2endifLatestObjectandContainer.Type=="Template"then--Only templates count as childrenLatestObject.Children[#LatestObject.Children+1]=ContainerendactiveObjects[#activeObjects+1]=ContainerelseifLatestObjectthenLatestObject:AppendText(Character)endelseifCharacter=="}"orCharacter=="]"thenifLatestObjectthenLatestObject:AppendText(Character)ifLatestObject.End==NNCthenifLatestObject.Type=="Template"thenLatestObject:Clean(true)finalObjects[#finalObjects+1]=LatestObjectelseLatestObject:Clean(false)endactiveObjects[#activeObjects]=nillocalNewLatest=activeObjects[#activeObjects]ifNewLatestthenNewLatest:AppendText(LatestObject.Text)--Append to new latestendendendelse--| or =ifLatestObjectthenLatestObject:HandleArgInput(Character)endendend--Step 4: Fix the orderlocalFixedOrder={}localSortableReference={}for_,Objectinnext,finalObjectsdoSortableReference[#SortableReference+1]=Object.Startendtable.sort(SortableReference)fori=1,#SortableReferencedolocalstart=SortableReference[i]forn,Objectinnext,finalObjectsdoifObject.Start==startthenfinalObjects[n]=nilObject.Start=nil--Final cleanupObject.End=nilObject.Type=nilFixedOrder[#FixedOrder+1]=Objectbreakendendend--Finished, returnreturnFixedOrderendlocalp={}--Main entry pointsp.PrepareText=PrepareTextp.ParseTemplates=ParseTemplates--Extra entry points, not really requiredp.TestForNowikiTag=TestForNowikiTagp.TestForComment=TestForCommentreturnp--[==[ console testslocal s = [=[Hey!{{Text|<nowiki | ||>Hey! }}A</nowiki>|<!--AAAAA|AAA-->Should see|Shouldn't see}}]=]local out = p.PrepareText(s)mw.logObject(out)local s = [=[B<!--Hey!-->A]=]local out = p.TestForComment(s, 2)mw.logObject(out); mw.log(string.sub(s, 2, out.Length))local a = p.ParseTemplates([=[{{User:Aidan9382/templates/dummy|A|B|C {{{A|B}}} { } } {|<nowiki>D</nowiki>|<pre>E|F</pre>|G|=|a=|A = [[{{PAGENAME}}|A=B]]{{Text|1==<nowiki>}}</nowiki>}}|A B=Success}}]=])mw.logObject(a)]==]