d5man/status(39) Language: English

D5Man Status Information

----------------------------------------------------------------------[ Meta ]--

name		d5man/status
section		39
description	D5Man Status Information
tags		note dev d5man work status
encoding	utf8
compliance	informal
lang		en
creation	2014/02/21
expires		2018/05/01 00:00:00
version		1.1.1
copyright	Copyright (c) 2014, 2015, 2018 Ma_Sys.ma.
		For further info send an e-mail to Ma_Sys.ma@web.de

------------------------------------------------------------------[ New Idea ]--

Evaluating the previous steps, it is clear that neither a Java-only, nor a
``decomposed Shell Script and C''-only solution is good. The decomposed solution
has shown to be too slow in conjunction with Ma_Zentral while the Java-only
version is far too complex and probably too heavy on resources as well.

Misc Conclusions
 * do not require attachment changes to be entered via standard interface
 * Sqlite selected, only use libsqlite, dumb server

Parts of the program not affected by the transition/no changes required
 * `d5man2xml`
 * `libd5manmeta`
 * `libmawxml`
 * `masitecopy`

-------------------------------------------------[ Board for new Development ]--


User Stories
 * export: text based figures (ditaa is written in Java!)
   ditaa works but outputs eps which needs to be converted to SVG and PDF
   to be useful in LaTeX and website contexts. This in turn requires that
   the external language processor can somehow invoke the media converter?
 * d5man2xml enhancement: creating a titled list w/ quotation in list title
   w/o leading space leads to obscure assertion failure (and invalid XML)
   `Test:"\`quoted"'` vs. `Test: "\`quoted"'`
    * make this produce a sensible error message (i.e. closing quotation w/o
      opening should cause immediate termination)
 * client: xml2d5man
    * requires the XML format to be final, i.\,e. most bugs in d5man2xml and
      such should be fixed _before_ attempting to work on this
    * especially re-think on description lists / titled lists!
 * replace IAL
    * The new substitute has to implement special functions `#...` using the
      API which has been established for the offline search.
    * Generally, it seems we do not replace IAL as a whole but rather split it
      into three parts
       1. Large API documentation (converted to d5man)
       2. Short information (converted to d5man or pdf)
       3. Medium-sized less important pages (e.g. the art of unix programming
          etc.) -> do not convert
 * general: tag MDVL packages by compliance for release
    * probably use directory trees and reprepro sections for this
    * requires full rebuild -> associated w/ MDPC 2.0
 * enhancement: descent/3(35) link download links
 * how to link blog entries (section 37) except if referenced from somewhere
   else? a page below the overview 35 section?
     * website/content: Decide how to process section 37. Shall we link from
       overview/thoughts(35) or from the news calendar only? (duplicates above)
     * use news via new Screenindex 2 (=> DLY)
 * website: Develop method to make PKGs available for download
    * Use existing system + a table of packages for which symlinks are
      automatically created/updated (simple and working approach!)
       * Won't play well with a repository but we do not have an online
         repository anyway.
 * create a page like a_case_against_online_activation 35 and link it from
   overview/mods(34) oblivion.
 * online_storage_comparison 36 could compare MyDrive GDrive and CX as well as
   Microsoft's storage (CX no longer accessible to the end user...)
 * create fusion_reactor/0 34 fusion_reactor/a 34 and ansichtskarte 34
 * website/usability how about SmartTV user agents? wrt. to max 30 cm width.
   -> Use media query
 * client: d5man as regular man
    * depends xml2d5man and man2xml (to be written or found somewhere)
 * general: tag filesystem using `xinfo.d5i` per dir
    * `d5mancompliant7z`
    * `d5mancompliantcp`
    * `d5mancompliantfind` (to be used w/ `cpio`)
 * New function: `d5mancompliantfind` (Java or C)
 * d5man(32) largely incomplete (_todo_ give a broad view etc.)
    * where to start -- start by giving a short description for all utilities
      => a nice diagram to show their interdependence would be cool.
    * which sub-pages will finally exist?
       * REQ: d5man(32), d5man/format(32)
       * Unsure: d5man/build_instructions 32 (how to compile on different
         platforms), d5man/setup 32 (configuration instructions, files...)
       * what else? Need a documentation for the LaTeX Template d5man/tex 32
         (esp. wrt offline server invoc w/ terminus, IAL generation etc.)
 * d5man2xml numbered items need to be prefixed w/ `0` to be equally wide.
    * This should not be necessary.
 * feature request: `gdi1/summary` could use a means of making titled sourcecode
   probably by adding an empty line indented with a tab to make it a
   ``description-type'' list.
 * Similarily, it would be useful to have a means of manually preserving a
   newline. In the spirit of D5Man we should just use `{\\\\}` for this :)
   (still need to recognize it in XHTML export!)
 * New function: D5Man Import Framework (to get IAL into D5Man)
   (pre-depends xml2d5man)
 * Website: Online Search function usage of temporary tables should now be
   deprecated in favour of real with statements which are now possible as
   per lima-city PHP update!
 * Website: HTTPs by default!
   A first attempt to implement a redirection did not yield positive results...
   (go for JavaScript w/ complete fallback and no issue wrt. moving content?)
 * Going backward in D5Man UI should position the cursor at the same place we
   came from => quick exploring possible
 * Create some wrapper scripts for common commandlines especially wrt to server
   and export invocation. This is like ``export by query'' or
   ``server by query'' or such. / Make sure these do not increase the time
   necessary to start common D5Man-Commandlines on Windows!
   -> Solved by creating a good `d5man` program
 * Allow `\\code{tikz}` to be used to mark following code as part of a TikZ
   picture (use this for D5 story diagram or such...)
 * Improve `d5manxml2tex.xsl` (see embedded TODO-markers)
 * LaTeX export: Make sure tables are sideways if too large
   (seems difficult to archive)
 * LaTeX export template: Assoc section numbers to description and display
   information on separator page. This pre-depends newly defining the meanings
   of sections which are currently already not-as-wanted.
 * Improve D5Man-TeX-Math-Interface to support dedicated quotations like in
 * Improve d5manvalidate to use parallel connections for weblink checking
 * Improve d5manvalidate to check for unwanted redirects for downloads
   (when pages have changed owner etc.) while retaining an ability to also
   accept ``wanted'' redirects
 * Improve (Windows) performance by reducing the path
   `d5man 39 d5man/status`. Generally we want to replace the current `d5man`
   by `d5manui`. Therefore, `d5manui` needs to support additional commandline
   arguments which allow other `d5man` program parts to be invoked, like
   `-x` for direct XML export to stdout, `-h` for HTML export by query, `-p`
   for PDF export by query, `-s` for server (by optional compliance)
 * d5man2xml does not handle `0x0c` (page break) in any useful means. It just
   strips such bytes out of the document. It would be good to add a tag to
   handle page breaks (this can be done in LaTeX as well!)
 * LaTeX export: `<<test>>` results in French quotation marks instead of two
   separate character on each side.
 * Creating google-drive links does not require an additional tool. It is
   sufficient to replace
   w/ `https://drive.google.com/uc?export=download&id=FILE_ID`
 * Import information about e-mail keys (non-convertable existing d5i and some
   HTML doc)... may be re-ordered to improve the process as a whole
   (ALT. create MDVL script)
 * D5Man Export: Triggering message in `d5manxml2tex.xsl` causes export
   to crash (wanted) but the corresponding message is not displayed (unwanted).
 * style should left-align text explicitly to avoid wrongly looking spaces.
 * d5man2xml: empty lines in code result in incorrect table indentation error
 * content: Ma_Zentral 11 contains german Archive Server descr,
   archive_server(33) the readme: MERGE!
 * document missing pre first try for mawebsite 3
   website/content: inconsistency mawebsite3/first_try(33) has a previous try
   which is not documented
 * d5man2xml fails if a titled numbered list with the first element `1. ~` is
 * d5manserver: Need to enclose text which begins w/ formula in `<p>` tag for
   correct indentation and validation
 * Invalid formula just results in empty space on website => Should produce
   a warning or error message during transformation.
 * d5man2xml: Slash between to code quotes not recognized as separator
   => only first part is processed as code, second part is reproduced verbatim
 * d5manview: Enter `Datenstrukturen und -Typen` and use `öh` and see
   that we get `Datenstrukturen und--Typen` (a superflous `-`)
 * d5manview: If window is about 80 characters wide saving yields unnecessary
   requirement to press enter.
 * d5manexport page w/ tilde for space just causes export to hang? -> ANALYZE
 * Potential regression in d5manexport: backup cxml pages fail to export to PDF
   package keyval tag undefined -- probably an issue w/ XML sourcecodes?
 * d5man2xml xml output indentation error: if `<i><math>` comes, `<math>` is not
 * d5man2xml: Code word and english quotation lack space between, consider
   `\`movs\` \`\`Move with sign extension''` (will lack space)
 * LaTeX export issue: `<<` is displayed as french qutoation, but we want the
   literal less signs to be displayed.
 * LaTeX export issue: all images displayed at 100% page width. Allow
   configuring only part of that (additional parameter to `\\img`)
 * d5man2xml: Staring a code with an ADA-Comment like `--` yields failure to
   parse this as a contra-list.
 * d5man2xml: Writing `Guards (\`::\`)\\n * \`...\`` makes d5man2xml produce
   the start of a description list inside the description list which causes
   a fatal error. Might be related to the other
   ``description list which contains list'' bug
 * LaTeX export issue: included graphics should be limited by page height
   (fgdi3 summary has an example of a graphic which is so long it does not fit
   on A4 export..., sort of `maxheight=pageheight` needed.)
 * d5man2xml: When processing a table with many smileys (cf. se/summary), the
   smileys in the last column are not displayed correctly => symbol resolution
   issue... not fixed completely yet.
 * d5man vim interface: Do not go back (on backspace) or forward on enter
   if file has open modifications
 * too many newlines in generated XML output (d5man2xml and Ma_Website!)
 * Bug: `D <<= k` results in quotation being displayed in D5Man LaTeX export
 * d5man2xml bug: \e results in in-symbol being dispalyed
   => remove this feature...
 * d5man2xml bug: description list content starting w/ `->` results in
   donwtriangle being produced instead of rightarrow
 * d5manvalidate displays a lot of errors wrt download fields missing in pages
   etc. which are false alerts... fix this
 * LaTeX complilation fails for source code w/ `€` character
 * d5manserver (or export?) creates a `.css` for `\\code{}` which is not what
   we want: We want to only save (and include) stylesheets when they are
   necessary, not when they are generated empty
 * LaTeX compilation fails for source code w/ `µ` character
 * d5man2xml math at begin of first table field not recognized properly.
 * d5man2xml bug sometimes dies during export of whole `internal` class.
   -> *** Error in /usr/bin/d5man2xml: free():
   invalid next size (fast): 0x0000000001053250 ***
 * LaTeX compilation fails for text which contains normal unicode greek
 * d5man2xml trailing `'` like in `... f'\\n ...` causes unexpected space error.
 * LaTeX export: ``What is GPGPU + Link'' from pmpp/summary 35 is not showing
   the necessary footnote.
 * xml2d5man: Need to encode backslashes for `<c>` tags.
 * LaTeX export: a list like `* \\[test` unexpectedly replaces the item's label
   and not the content. Template should generate a `\\relax` before first
   user-supplied characters.
 * LaTeX export escaped backslash (textbackslash) in D5Man has no trailing
   space even if one was present in the D5Man source code (due to the
   necessity of generating a reactor symbol after textbaslash to treat it as
   a trailing space)
 * d5man2xml: code inside `<` and `>` characters is not produced correctly:
   `<\`test@example.com\`>` does not print the e-mail in teletype font.
 * Website bug: web/experiment/ein_mal_eins(31) exported to invalid XHTML
   (header is wrong)

In Progress

User Stories
 * server: write newspage
    * it was decided to do this using Screenindex 2 => do not wait for this
      before publishing the new website.
    * In the meantime it is advisable to create a placeholder like newsp 31
 * Re-Merge Description Lists for LaTeX (and XHTML as well) -> Amend d5manxml...
    * Developing this feature seams incredibly expensive esp. wrt. all the
      dependent applications which need to be changed (this can not be achieved
      sanely within the current XML API)
 * Now working on d5manimport (needs to be re-written in Java)
 * currently not working on any defects


User Stories
 * Replace old D5Man w/ new implementation already?
    * Reintroduced `d5man` script for user invoc
 * Check if XML can be used as a data exchange format for local websites (OK)
 * Split off `libmawxml`
 * Transition `d5mandbinit`
 * Create `d5manquery`
 * Design and write `d5mandbsync`
    * Take inspiration from old `d5mandbsync` :)
    * Idea: Generally we sync taking pages from stdin (potentially multiple
      zero-separated pages like w/ `d5man2xml`)
    * In order to handle attachments, we rely on `d5mandbsync -a` to update
      attachments by obtaining a path using `libd5manioresolve`
      create io resolve lib
    * Downloads not recognized
 * Transition `d5mandbcompliancedup`
 * Design and implement `d5manioresolve` Commandline Utility
    * This is an interactive resolver (line by line from stdin)
    * We need to define a simple protocol/format for ``answers''
      (generally, CSV might be a good format for an answer / alt. very simple
    * IDEA: Output is CSV and commands can be used like old `d5manio` (i.e.
      `-r` and `-w` parameters are added in front of the page depending on the
 * ``Writing to commands'' is necessary in d5manioresolve. We therefore need to
   define how processes are invoked differently for r/w and where the difference
   is actually considered...
 * Rewrite `d5manview.vim` and related
 * Update `d5manui` to use new viewing mechanism and to directly invoke VIM
   (needs some logic to create new pages!)
    * Need to extend the new `d5manview.vim` to be able to create new pages
      if invoked w/ correct function. (OK)
    * Extend DB to be able to produce names for result codes of resolve queries
    * Allocate currently stack'ed variables to heap and create new variable on
      stack to hold the commandline (OK)
 * Implement `d5mandelete [-d] section name` to delete pages
   (`-d` means from DB only)
 * ... Establish Java Library `libd5manexport`
    * Contains APIs to communicate via `stdin/stdout` w/ D5Man C parts
       * Also abstracts/supports process pools for quick responses
    * Contains APIs to parse the result of these processes
    * Contains general classes like `D5Stream` and similar
    * Aim at creating functions as listed below
    * XML processing taken from second try for server
 * Website: Create layout/html templates for ``download boxes'' and add server
   code to support doing this.
 * Website/Server inconsistency: If `\\code{lang}` is used once, every code gets
   a black ``window'' / if it is never used, the code is just printed on the
   page. Dow we want to keep it this way or prefer all black windows?
    * Probably use `#500000` (link background) for all code boxes
      (except for page source code: fill whole page black)
 * Website: Extend Template to contain links to the source of a page and back
 * Website Content: Extend existing pages to contain download fields
    * Also add suitable links from overview pages
    * Also there should be backlinks from the download boxes to their origin
      pages (need to amend DB query!) This is HRC for perfect website
    * Check if download page generation works (OK)
 * Sitemap should list all `@src.xhtml` pages with ultra low priority (one tenth
   of original page priority?)
 * ... Implement D5Man Server
    * attempt a full website upload after backup
 * Important enhancement: Before attempting to write `d5manexport` we need to
   merge the ``pages'' created for the server into the `XHTMLExport` and
   review `ProcessingResult`. Finally, the `page` package should be below
   `xhtml` and only one `XHTMLExport` should be all the server and export need.
    * Which of the parameters to `XHTMLExport` should be created inside that
      object (i.e. are never shared)?
      Parameters: `Conf`, `D5ManDB`, `DownloadsTable`, `D5ManIOResolver`
      (new interface to abstract from processes), `D5Man2XMLProcess`
      Created by Export: `VIMProcess`, `ExternalLanguageProcessors`
    * Should `XHTMLExport` really close potentially shared resources?
      (no it should not)
    * General idea: Allow multiple `XHTMLExport` instances to operate
      concurrently sharing some of their processes to reduce resource usage
      maintaining a high speed of processing for parallelized exports to be
      implemented later.
    * Create `XHTMLTemplateValues` to create the entity replacement table from
      a given `AbstractD5ManDBPageMeta` and content Strings
       * Use this to process `ProcessingResult` as well as `DDownloads`
    * Move `DataSource` to `ma.d5man.lib.export.xhtml.pages.` move `md5` to
      `D5Stream`. Then link them together in `XHTMLExport` and make sure
      necessary objects are managed by the `XHTMLExport`
      need to implement all the `getMD5` methods
    * The previous `DMan` is merged into `XHTMLExport` so that `XHTMLExport`
      implements `MinimalResourceSet` / or shall we create a method like
      `XHTMLExport.viewAsResourceSet()` to return something like `DMan`
 * LaTeX Template / General functionality OK
    * Develop a nice design OK
    * Attempt to compile all pages OK
    * Code-Language Lookup Table OK
    * Add TOC OK
    * Add a custom front-page (might depend on the compliance level)
      (need to be designed as well) EWONTFIX / idea superseded ... one page is
      too much we just create a notice at the begin of the ToC
    * distinguish between multi-part/single-part and single-page exports
      (multi-part has separate parts, single-part not and single-page does not
      have a TOC). Also, exported pages should appear in a sensible (alphabetic)
      order grouped by section (and those should be ordered ascending) OK
 * client: LaTeX converter
 * ... New functions: `d5manexport` especially TeX export (Java)
    * System critical issues OK
       * Circular exceptions
       * Exception causes whole program to halt
    * Other question: Why are there errors in the first place? OK
      (due to this file being misformatted)
    * Progress display OK
    * Downloads table not exported correctly OK
 * Missing export formats LaTeX Export
    * Need to generate all-including overview and static template
      (also do not include dtd in output as it is meaningless here /
      similar for XHTML export!) OK
    * Need Template and what about `\\img{a.svg}{Ma_Sys.ma}`
      => need to replace svg/pdf and `_` / `\\_`. The whole thing is not
      very nice... find a good solution... OK
    * Idea: Replace `pdf/svg` in LaTeX and `_` need to be escaped in
      the first place => amend XHTML epxort to unescape them... OK
    * Template problem: Does not support leading and tailing spaces...
      cf. lstlistings documentation... OK
    * Sort pages before LaTeX export OK
 * Use `d5manexport` instead of `d5manserver` for Website generation purposes.
 * New function: It would be good if D5Man's utilities (like export and server)
   could also act on detached pages in order to be able to still use sort of
   "`no longer relevant"' but still existent pages. This category would be most
   useful for readme-files for solutions to exercises and notes tied to some
   other non-website (or print-only or hardware-related) topic. Currently, there
   seem to be two means of implementing this
    1. Amend programs like export and server to be able to take files directly.
       This makes it easy to export single pages and has little overhead
       although the implementation is rather difficult
    2. Create scripts to perform live-creation of necessary meta-structures like
       DB and configuration. This has a large overhead but is least work to
       implement and also works with hyperlinks and multiple pages as well
       as attachments and downloads tables etc.
 * Use Terminus font for sourcecode online
    * Download terminus fonts
    * Use `fontforge` to convert them properly
    * There is a TTF for the font -- no need to convert :)
    * Probably not a good idea wrt to printing
    * also think about cancelling all webfont usage (except for titles?) because
      it makes page rendering strange and slow SEL
 * ... New Functions: `d5manvalidate` (Java)
    * add: Validate to check if the current page converts to XML properly
      and if all attachments are avaliable. Create a `d5manvalidate` for that
       * This could improve the checking script as well.
    * create a link checker to check all pages references to other pages as
      well as weblinks by evaluation the HTTP response. `d5mancheck [query]`
       * we also need to be able to reverse-search for links upon renaming
         pages etc.
    * It should be possible to also ``validate'' if the page can successfully
      exported to valid LaTeX and XHTML? How would we access proper exporting
      for this? move more functionality to `libd5manexport`? -> This works now
    * As it stands, `d5manvalidate` highly depends on `d5manexport`. It might
      be advisable to just add it to the same source-tree and just make
      validation perform the following three general steps (all optional)
       1. Convert to XML, Check w/ DTD
           * Check internal and online links -> OK
           * Browse the FS for unlisted files and compare db metadata w/
             reality (as in existing partial implementation) OK
       2. Convert to XHTML, Validate XHTML (using XHTMLExport) (TEST)
       3. Convert to LaTeX, check if it compiles (using LaTeXExport, TEST)
    * create status thread and merge all validation tasks which use
      an XML export (perform file export and then validate the files?) OK
    * Check if validation works / We say it's OK / Failures below / Rest is OK!
       - Wrong report for `/data/main/man/d5i/38/d5_clock.d5i` (``download
         main missing in D5I file header'') ... was not in sync w/ DB?
       - Unclear if LaTeX export works ... watch log details...
 * se/summary (35) sometimes two newlines, sometimes one between descriptions?
 * d5man/format(32) not exported to XHTML (`NullPointerException`)
 * Validator links broken in new server!
 * `db_io_resolve.c` needs to create missing directories if asked for
   writing to real files.
 * Depending on the window size the server is running in, exported code has
   line numbers or not. export environment vairable to make sure code never
   shows line numbers (or do something like `set nonumber` or such)
 * Website VIM Code has ID instead of class (and other properties than normal
 * `d5manmirrorautoftp` uses old invocation `d5man -s` => defunct till fixed
 * server/website: MathML not embedded correctly, validator says like there is
   no attribute `xmlns` for `math` and no MathML elements exists
   gives like 2000 errors for GDI II page
 * d5man export race condition appeared: did not terminate w/ two processes
 * `index.xhtml` does not redirect in `d5man -s`-Mode
 * d5man2xml does not work w/ Windows newlines (`\\r\\n`) but in order to be
   able to process Oblivion-Mod Readmes, we need to support this.
 * Threading problem `d5manexport` hangs at exporting `mss_summary(35)`
   (see text file `todo_threading_problem.txt` / probably related to assertion
   failures e.g. caused by `{$\\eta}$` -- an invalid d5man2xml input...)
    * Found out that the `StderrCacher` does not notify the result buffer
      thread correctly (due to the `StderrCacher` already having the output
      before a thread is registered to be notified) ?
      -> Correct StderrCacher.
    * `StderrCacher` was not the issue. The terminator was not sent correctly,
      because it appeared not indented as necessary => TODO MAKE SURE THE
      TERMINATOR IS ALWAYS INDENTED ... TEST (-> a d5man2xml defect?)
 * web/about(31) Key "`Online-Verfügbarkeit"' Link defekt -- wieder behoben
 * Important regression introduced by switching to new XML-list format
   Incorrect indentation (titled vs. untitled lists) in Ma_Sys.ma Website 4
   (potential solution: do not indent all lists inside first level because
   indentation is same regardless of title/no title...)
 * LaTeX export: If lang=de the ToC is still called ``Contents'' but should be

--------------------------------------------------------------[ Dependencies ]--

 * php5-sqlite, dash, php (+ webserver), vim, libsqlite3, libncurses
 * libsqlite3-dev, libncurses-dev

To be checked
 * `ditaa` (diagram drawing) to be able to include diagrams in D5Man pages
 * Which LaTeX packages are necessary to build a new Ma_Sys.ma PDF?

Portability Issues
 * `libd5mandb`: `d5man_mkdir.c` relies on UNIX directory manipulation.
   (can be solved by using `system` or an OS specific directory
   creation command / error reporting need not be correct)
 * `d5manui`: `d5manexec.c` relies on UNIX `exec` function
   (this can be solved by simply using a `system` call)

----------------------------------[ Ideas from a complex Java implementation ]--

Solving database problems
 * Our new program is able to contact two databases
 * There is a user-database `~/.mdvl/d5man/db` and a system database
 * Upon installation of a package which contains a d5man-file for documentation,
   this file is processed through `d5mandbsync` which then updates the system
   database at installation times.
 * When the user requests a page through a query, pages from both databases are
   considered, but pages from the local DB override pages from the system db.
 * This allows us to use MDVL normal and also have an installable version of all
   pages we also locally have
 * This allows users to edit system pages by saving a local copy
 * The only situations which is not supported:
   If we create/edit a local copy and then the page gets updated, we still have
   the old version! This could be mitigated using a special validation utility
   checking for newer system pages than local pages with the same name!

Configuration Parameters

Include default configuration resource to be consulted first in order to
have some sane defaults for all values... (?)

	root_shared=<Shared data write root>
	root_personal=<User data write root>
	ro=1 # optional (this is for ``read only'')
	# Combine these with any number of db, excatly one jdbc and optionally
	# more jdbc0, jdbc1... (all from high to low priority)

--------------------------------------[ Note System and Website Requirements ]--

Note System
   a) wkhtmltopdf for short lived things
   b) print editing, save html and then wkhtmltopdf
   c) use text2html to convert html directly to text (looks acceptable)

	  Different methods of storing
	  Method            Descr
	  D5-Manpage        completely edited/reformatted html2text
	  Plain Text        slightly edited html2text
	  PDF               from wkhtmltopdf
	  ZIP of Scrapbook  if nothing else helps

Transitioning is _very_ important... items should be able to be moved till they
are respected D5 Manpages or such. We have introduced the `expire` attribute for

online: `<!-- result -->`
offline: `<script type.../...>`

	<!-- <?php /* -->
		<script type="text/javascript" src="offlinescript.js">
	<!-- */ require("....php"); ?> -->

Website Requirements
 1. Static X/HTML must work.
 2. Periodic Login must not be necessary.
 3. Basic PHP must work. Statistics will be collected using a Write Ahead Log
    text file which will then be evaluated on the ``source'' system.
 4. FTP or WebDav access is required to programmatically upload files.
 5. Basic HTACESS Access management to hide DB and statistic files.

---------------------------------------------[ Board before the third review ]--

(only items not listed above; ideally all of this is solved once the third
transition has been completed)


User Stories
 * website: improve usability
    * add `download` custom field which specifies link,chcksm,file or such and
      automatically generate the downloads overview page w/ smaller text for
       * This is a similar issue to the sitemap generation. we need additional
         fields there, too
         => change DB (add some columns) or rewrite in Java using another DB?
    * add a ``download'' button to project pages for which downloads exist
      (make a little box w/ chcksm, size, download link etc.)
 * wesite: develop a means of viewing the page sourcecode online (ideally w/
   syntax highlighting enabled) for all pages which do not include dynamic
   PHP. This needs to be implemented in the d5manserver
    * Use `:TOhtml` VIM command to solve part of the problem (the result has to
      be pre- and postprocessed to generate compatible and hyperlinked output)
 * server: higlight sourcecode
    * The VIM Interface has been written but it is necessary to connect it to
      the server properly. _XSTAT_ Where to do this?
       * To begin with, we need to be able to read `d5manio -r` directly
         to be able to display page sourcecodes. _PART_
          * Some restructuring has been started in a separate ``fork'' below
            `xdev/test`. Continue there.
       * Next, the template needs to be shared among page source code and
         the page generator in order to be able to access the menue while
         viewing the page sourcecode.
       * Finally, the template needs to be changed in order to be able to switch
         between src and not src (probably in the footer or header?)
 * d5manmove fails to populate and delete attachments
 * d5man2xml fails for words with tailing `~`. Is this a bug?
 * d5man2xml does not properly indent XML for tables which contain a single
 * d5man2xml regression does still not handle english quotation at begin of line
   correctly (cf. gpl page)
 * d5man2xml does not handle `M-S--` in a table correctly (fails w/ unexpected
   space). Cf. spectrwm_kbd(42)
 * d5manserver Website sourcecode XHTML export not working since Debian 8
 * Description list entitled w/ value in quotes fails w/ OOM
 * Defect in `d5mandbsync` (see `Makefile`)


 * d5man2xml: does not support nested quotation
    * could not reproduce so far. only occurrs under specific circumstances?
      or resassign to d5manserver?
 * website: Search results contain `search.xhtml` page at high positions because
   of missing sitemap for Google
    * A rough attempt has been taken hardcoding important page names in
      `Sitemap.java`. This, however, can not be considered a final solution.
    * Ideally, we would add some optional meta values which could then be used
      to identify ``change frequency'', ``relative importance'' and potentially
      also a `<meta name="description".../>`
       * This is related to the usability storie's ``download buttons'' idea
         which also requires some additional fields.


User Stories
 * d5man2xml: Add special ``math mode'' `{$` and `$}`
    * server: implement math mode correctly
 * client: write
 * server: compliance / data stores sensitive parts and is not released in
   source form. Other sensitive parts just get compliance `internal`
 * link experiment `ip.php`
 * js_info(33): add this script to the local experiments section to make it
   available online. File created but has to be linked to.
 * chartable(33): add this to the local experiments section...
    * Problem: could be abused to drain performace. Limited to 20\,000
 * ein_mal_eins(33): add the script to the local experiments section to make it
   available again. and link
    * configure stylesheet properly (buttons etc.)
 * website/design
    1. Navigationsleiste Rotton schwächen (cf. aktuelle css)
    2. Brauntöne versuchen (fail)
    3. Textbreite ab 1280px incl (!) zu groß -> max 30 cm Textbreite
 * server: mirror the page by compliance: should the server hide any acompliant
   parts or should we mirror the db and cnt and start the server from it?.
 * content: convert Ma_Sys.ma PDF
    * work on page tracking
    * content: add new Projects: dirstat 2
 * mirroring the page needs an incremental FTP utility
    * ftp: Try `wput` or `ncftp` (has an automated put utility as well).
    * ASEL `sitecopy` which is in Debian and possibly available for Windows
      as well + a simple wrapper script.
 * server: test 404
 * d5man2xml: Make `(!)` a symbol
 * d5man2xml: Make [CTRL]-[C] the markup for keyboard shortcuts
    * In order to complete this, we will need to update the server to be able to
      process the new tags.
    * implicitely corrected defect: server displays Ma_Zentral 12 shortcut table
 * server: The generation of offline search indices is TBD
 * server: the search function (see below)
    * php part almost ok except for output layout (how about asking the lima
      city forum?) -- did not get any better suggestions so far.
    * make the js part replicate the php part as good as possible using offline
      search indices...
    * Search index format `/pagedb.js` w/ JSON.
      main -> section -> list of page names
      tag -> tagname -> list of page section and name for this tag
    * need to be able to search by tags. We need to manually perform the
      join in SQLJet because the library does not do it for us.
    * next, the ``client side'' javascript part is to be implemented.
    * finally, we want to be able to navigate in the search just like w/ IAL.
 * d5man2xml: permit things like `a--b` for a to b.
 * Set correct viewport and larger font size as well as larger button distances
   (?) for mobile devices ... -> Google Webmaster Tools recommends to do this.
 * d5man2xml table in d5man/format(32) contains four dots where three should be.
 * d5man2xml does not create pragmas w/ `{<` and `>}`.
    * This has been corrected => implement changes in server
 * d5mandelete utility does not exist
 * d5manexpire defunct (news should be expired)
 * d5manserver: process output does not become accessible w/ waitFor if it
   exceeds some internal buffer
    * d5/main(38) never returns
 * design: consider using yellow links instead of purple ones?
 * server: 404 includes wal two times
 * server: news includes default two times
    * E couldn't reproduce
 * package: perform db compliance dup upon package upgrade; program or data pkg?
    * data pkg makes more sense but program pkg is easier to implement
 * server: quotes not localized properly in Iceweasel
 * d5man2xml fails to create links in tables properly (cf. web/downloads(31))
 * MAZENTRAL backspace does not work on Windows
 * website 404 sometimes fails to work correctly.
 * content: add archive server readme to d5man system
 * d5man2xml produces a space after opening code for tools/1(33) tables
    * related to the fact, that a proper 0-separator is not placed...
 * d5man2xml: does not support quotation at begin of paragraph. Workaround:
    * prefix `_{}_`
    * REASSIGN to D5Man Server (d5man2xml is not responsible)
 * d5man2xml creates ``in'' symbol if not surrounded by spaces
   (cf. file_transfer(35))
 * d5man2xml does not handle `...` at the begin of a first field inside a table
    * This is due to some leading spaces being present.
 * d5man2xml fails to handle empty table field as in messgeraete(35)
    * potential solution: Define "`empty"' symbol or recognize `~` in TBL
       * The reason this is not recognized currently is that `~` is only
         a space if in mode `INWORD`. Otherwise, it is considered a normal word,
         which does not apply here.
    * alt: document to use empty code symbol `\` \``
 * d5man2xml fails with code in description list title
    * Spaces get lost unconditionally (cf. gdi1)
    * The problem occurs once normal parenthes are present because
      `parse_preprocess.c` recognizes the input as a link and therefore
      removes all spaces. => we need to add a `MODE_PREPROCESS_CODE` to skip
      over code during the preprocessing stage
 * Website stylesheet fails on Chromiums device simulation mode (much too large)


User Stories
 * website/program: content security policy
 * regex(33): add this to the local experiments section to make it available
    * `/e` is to insecure => not added
 * website/misc: Add lima-city logo in the bottom line if mirroring to lima-city
    * add a server option to allow the user to insert additional banners at the
      bottom of the page or at least add an option to display one additional
      banner for the "`hoster"'
    * A simpler solution has been chosen: The banner is now always displayed
      (who cares if offline IAL also advertises Lima-City?)
 * content: make start page special for IE on windows 8
    * url(http://msdn.microsoft.com/en-us/library/ie/bg183312%28v=vs.85%29.aspx)
    * `<meta name="msapplication-navbutton-color" content="#ff0000"/>`
    * Did not work as expected => We leave this task to the OS.
 * D5man Format: Think of allowing the ``new'' LaTeX math notation which uses
   `\\(` and `\\)` for inline math and `\\[` and `\\]` for separate formula
   (we'd override `\\[...\\]` to support multiline (`gather*`) equations of
   course). We could use this to support math explicitely on websites and
   reserve the LaTeX `{...}` for real LaTeX? EWONTFIX
 * Windows build script
    * Check if adding flags removes mingw dll dpendencies (-> sqlite build)
 * content: Grutatext website linked from d5man/format(32) no longer exists.
   Some recovery has been performed using the Wayback machine and Google Cache.
   (Resolved by the site appearing again)

---------------------------------------------------------[ Status 2016/06/07 ]--

... as reported by `cloc` ...

	Language         files          blank        comment           code
	Java               211           2204            831          10006
	C                   62           1058            554           7617
	C/C++ Header        70             99            253            727
	XSLT                 3             83             56            643
	make                27            102             17            446
	Bourne Shell        10             89             37            386
	vim script           3             43             82            221
	DTD                  2             34             38            111
	SUM:               388           3712           1868          20157

Zum Seitenanfang