Wikitech-l April 2008

wikitech-l@lists.wikimedia.org

100 participants
99 discussions

Database dumps
by Byrial Jensen 17 Apr '25

17 Apr '25

Until some weeks ago http://dumps.wikimedia.org/backup-index.html used to show 4 dumps in progress at the same time. That meant that new database dumps normally was available within about 3 weeks for all databases except for enwiki and maybe dewiki where the dump process due to size took longer time. However the 4 dumps processes at one time become 3 some weeks ago. And after massive failures at June 4, only one dump has been in progress at the same time. So at the current speed it will take several months to come thru all dumps. Is it possible to speed up the process again using several dump processes at the same time? Thank you, Byrial

3 2

EBNF grammar project status?
by Steve Bennett 01 Apr '25

01 Apr '25

What's the status of the project to create a grammar for Wikitext in EBNF? There are two pages: http://meta.wikimedia.org/wiki/Wikitext_Metasyntax http://www.mediawiki.org/wiki/Markup_spec Nothing seems to have happened since January this year. Also the comments on the latter page seem to indicate a lack of clear goal: is this just a fun project, is it to improve the existing parser, or is it to facilititate a new parser? It's obviously a lot of work, so it needs to be of clear benefit. Brion requested the grammar IIRC (and there's a comment to that effect at http://bugzilla.wikimedia.org/show_bug.cgi?id=7 ), so I'm wondering what became of it. Is there still a goal of replacing the parser? Or is there some alternative plan? Steve

26 217

MediaWiki to Latex Converter
by Hugo Vincent 18 Jun '12

18 Jun '12

Hi everyone, I recently set up a MediaWiki (http://server.bluewatersys.com/w90n740/) and I need to extra the content from it and convert it into LaTeX syntax for printed documentation. I have googled for a suitable OSS solution but nothing was apparent. I would prefer a script written in Python, but any recommendations would be very welcome. Do you know of anything suitable? Kind Regards, Hugo Vincent, Bluewater Systems.

6 13

Replacement stats for placeholder images?
by David Gerard 13 Oct '09

13 Oct '09

I've been putting placeholder images on a lot of articles on en:wp. e.g. [[Image:Replace this image male.svg]], which goes to [[Wikipedia:Fromowner]], which asks people to upload an image if they own one. I know it's inspired people to add free content images to articles in several cases. What I'm interested in is numbers. So what I'd need is a list of edits where one of the SVGs that redirects to [[Wikipedia:Fromowner]] is replaced with an image. (Checking which of those are actually free images can come next.) Is there a tolerably easy way to get this info from a dump? Any Wikipedia statistics fans who think this'd be easy? (If the placeholders do work, then it'd also be useful convincing some wikiprojects to encourage the things. Not that there's ownership of articles on en:wp, of *course* ...) - d.

7 11

GSoC Project: Category Moving
by Tim Johansson 29 Jun '08

29 Jun '08

Hello, wikitech. I have applied to Google Summer of Code with the project to enable category moving without using bots. After some correspondance with Catrope, the following text is my project idea. Any feedback would be welcome. Synopsis I will provide capability of moving categories to achieve an effect for the end-user similar to that of moving other pages. Currently, contributors must apply to use a bot that recreates the category page and changes the category link on all relevant articles. Project The object can be divided into three parts. First, the category page is moved, along with its history, just as renaming of articles works. A redirect is optionally placed on the old category page, and the category discussion is moved as well. Second, all articles in the relevant category must have their category links changed. There are several obstacles involved in this task: 1. Finding all alternative ways of categorizing articles. It is simple to match the simple category links and category lists, but more difficult to find e.g. categories included from a template. Roan Kattouw (Catrope) suggested category redirects for this, such that all articles categorised as [[Category:A]] would also be listed at [[Category:B]] if the prior has been redirected to the latter. 2. Articles might be in the process of being edited as the movement is done. This, however, can be solved in the same manner as edit collisions are currently solved. 3. The algorithm would likely have high complexity and would thus not scale well with very large categories. This is likely to constitute a significant and challenging part of the project. As the last step, the relevant entries in the categorylinks table would need to be changed. This is accomplished by a simple SQL query. This could be avoided if bug #13579 [1] ("Category table should use category ID rather than category name") is fixed, which it could be as part of this project. The project would preferably be written as a patch to the core. Catrope suggested setting up a separate SVN branch for the project, such that everyone can see my progress. Profits for MediaWiki Developing a means of moving categories would decrease dependency on bots, gaining in administrative time. Additionally, the solution would be faster than any bot-relying solution could be due to, among other things, the removed need of loading pages. Category moving would also increase the consistency in layout on the different article types. The only real reason for a "move" tab not to reside on category pages is that the feature is not yet implemented. Roadmap Publishing this document to the MediaWiki development community (wikitech-l) and awaiting comments on the planned procedure would be the first step. After the community bonding period specified by the time line, a week should be enough to get comfortable with the relevant MediaWiki code and implement the first section, moving the category page along with its discussion and history. Much old code should be reusable here, such as the Title::moveTo() method for moving pages. Until mid of July, most of the second part of the project should be finished. In a week from there, the last part would be completed, too. A month is then reserved for bug-testing, tweaking and as a buffer for unexpected obstacles. The MediaWiki community is very important in this step for testing and feedback. Regards -- Tim Johansson http://timjoh.com/

8 13

Case insensitive links (not just titles).
by subscribe＠divog.com.ru 23 Jun '08

23 Jun '08

Hi Sorry for my English :) What I need is case insensitive titles. My solution for the problem was to change collation in mysql from <unf8_bin> to <utf8_general_ci> in table <page>, for field <page_title>. But bigger problem with links persists. In my case, if there is an article <Frank Dreben>, link [[Frank Dreben]] is treated like a link to an existent article (GoodLink), but link [[frank dreben]] is treated like a link to a non-existent article, so, this link opens editing of existent article <Frank Dreben>. What can be fixed for that link [[frank dreben]] to be treated like a GoodLink? I've spent some time in Parser.php, LinkCache.php, Title.php, Linker.php, LinkBatch.php but found nothing useful. The last thing I tried was to do strtoupper on title every time array of link cache is filled, in LinkCache.php. I also tried to do strtoupper on title every time data is fetched from the array. I've tried to make titles in cache be case insensitive, but it didn't work out, not sure why - it seems like when links are constructed (parser, title, linker, etc) only LinkCache methods are used. Could anybody point a direction to dig in? :)

7 36

confirmemail_body and create account
by Andreas Rindler 23 Jun '08

23 Jun '08

Hi, I need to create some user accounts and have mediawiki send the users a welcome message. I found the Mediawiki:confirmemail_body page which appears to feed such a welcome message. How do I do that? If I use Special:Userlogin, create an account and use "by email", the user gets a password reminder, but not a welcome message. I could change the password reminder text, but that wouldn't be correct for all the other users who use that page to genuinely retrieve their password. Thanks, Andi

3 5

Re: [Wikitech-l] Wikitech-l Digest, Vol 57, Issue 25
by Aerik Sylvan 26 May '08

26 May '08

S Page wrote: > Aerik Sylvan wrote: > > At the risk of asking a stupid question: what is the status of category > > intersections? > > At the risk of making a stupid answer: you could install the Semantic > MediaWiki extension and make queries like > > {{#ask: > [[Category:Actor]] > [[Category:Director]] > }} > > SMW will query for membership of subcategories (thus it'll match members > of Child actors) , to a configurable depth limit. The nifty thing is > you can > display other properties and categories of matching pages. > > See demo (temporarily) at, > http://www.semanticweb.org/wiki/Sandbox#Category_intersections > > That's cool... what's the backend for that? (I looked briefly at some docs at http://semantic-mediawiki.org/wiki/Semantic_MediaWiki but didn't quickly find any architecture stuff). Best Regards, Aerik -- http://www.wikidweb.com - the Wiki Directory of the Web http://tagthis.info - Hosted Tagging for your website!

2 1

Wysiwyg
by Cormac Lawler 23 May '08

23 May '08

Hi, I know a lot of people would love wysiwyg editing functionality, I know a lot of people on this list are working on such a thing, and I know that a problem seems to be the "grammar" involved. I heard on the grapevine that, on this basis, it was "at least a year away". I don't want to know the specifics of what that grammar is, or what its problems are. :-) But what I'd like to know is: * Would this issue be solved quicker by having a few funded developers working on it? (If so, how many wo/man hours would be needed - what size of a funding bid would be needed?) * What does the fckeditor in existence *not* do yet? < http://mediawiki.fckeditor.net> (If the answer is: "too much - it only does a tiny handful of things compared to what we need" - well then, that answers my question. :-)) If you haven't already surmised, this question is from a technical dunce - answers at that level would be greatly appreciated. :-) Cheers, Cormac

5 4

Re: [Wikitech-l] So... status of category intersections?
by Aerik Sylvan 22 May '08

22 May '08

On Tue, 22 Apr 2008, Simetrical wrote: > > On Tue, Apr 22, 2008 at 10:59 AM, Roan Kattouw <roan.kattouw(a)home.nl> > wrote: > > I missed the explanation of the fulltext implementation. Something like > > 'Foo With_spaces Bar' and then do a fulltext search for the cats you > > need? That would be more powerful, and would probably be faster for > > complex intersections. I'll write an alternative to > > CategoryIntersections that uses the fulltext schema and run some > > benchmarks. I expect to have some results by the end of the week. > > Aerik Sylvan has already done an implementation of the backend using > CLucene. If a front-end could be done in core, with a pluggable > backend, that might have the best chance of getting enabled on > Wikimedia relatively quickly. MyISAM fulltext is not necessarily > going to be fast enough due to the locking. > > Yes, I did a fulltext search (which works quite well - I forget the response times... I think it was around a third of a second even for intersections of large groups, like "Living_People") and the way it handles booleans and stuff is quite nice. I think I broke it when I moved servers, but I can put it back up. I think it would probably be a great addition to core, and would be very adequate for small wikis, but too slow for larger ones (performance at a few tenths of a second will really add up with tens or hundreds of hits...) I think doing updates is also an issue on large wikis, due to table locking of the MyISAM table. But, I think it will be fine for small wikis. MySQL doesn't break on underscores, so using the category as it appears in the url seems to work great for fulltext search, and the built in fulltext search is *much* faster than doing lookups on the categorylinks table, especially for large sets. So, I'd propose in core we add a MyISAM table with a fulltext index of categories - this will suite small wikis. For big wikis, make this a InnoDB table and use it to build a Lucene index, which you'd search with whatever flavor of Lucene you like. This is a fairly straight path, that covers both core and large wikis, should have good performance for either application, and is flexible in that it does boolean searches. I don't have suggestions for an interface, but why not just start with a SpecialPage and see what happens? Once the functionality is there, suggestions for how to better use it will come out of the woodwork. I'm working on a CLucene daemon (calling it clucened, which is on SF - with slightly out-of-date source in subversion - and at clucened.com), which could be used for this, or anything else. I'm planning to make it Solr compatible, but not a direct port of Solr, and the implementation will have some differences. So far I have only the daemon and the search function (takes a raw query, which can be boolean or have mulitple fields, and passes it through). I think this is really cool, but if we already have a GCJ Lucene search for En, it may be easier just to extend that to read at categories Lucene index than use another architecture. Either way, I think a search daemon will find an audience and will be a really cool thing :-) Aerik -- http://www.wikidweb.com - the Wiki Directory of the Web http://tagthis.info - Hosted Tagging for your website!

8 14

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l April 2008