firefox_extension_modernization_issues(37) Language: English


From Firefox 52 to 60: Dealing with the Deprecation of SDK/XUL Extensions

----------------------------------------------------------------------[ Meta ]--

name		firefox_extension_modernization_issues
section		37
description	From Firefox 52 to 60: Dealing with the Deprecation of
		SDK/XUL Extensions
tags		firefox extension mdvl blog
encoding	utf8
compliance	public
lang		en
creation	2018/09/09 12:40:08
version		1.0.0
attachments	httpfox.png noscript.png classic.png vimfx.png wcol.png
		nocol.png scrapbook.png userstyle.png ext60.png ext52.png
copyright	Copyright (c) 2018 Ma_Sys.ma.
		For furhter info send an e-mail to Ma_Sys.ma@web.de.

--------------------------------------------------------------[ Introduction ]--

Ignoring all the different variations and forks (simplifying the matter a lot),
three different rendering engines for desktop web browsers exist. Representative
for the different engines are Google Chrome, Internet Explorer and Firefox.

On Linux, Internet Explorer is not available and thus there are two major
choices: Google Chrome and Firefox.

Here, Firefox was first used in version 2.X (at that time still on Windows) when
Chrome was not yet available (Konqueror with the KHTML rendering engine was).
Since it worked well at the time, Firefox was kept and upgraded to new versions
whenever needed.

As Google Chrome (and the open source Chromium) emerged, the were also tested
but two major properties of Firefox dismissed them as alterantives:

Firefox Extensions
   Very early, Firefox supported powerful extensions already and a lot of useful
   extensions were found over time which could not adequately be replaced by
   Chromium-compatible ones.

Browser Behaviour
   Different rendering engines have different behaviour. Two major things which
   were found better in Firefox were the handling of selections and the
   compatibility with internally-developed local sites (Chromium forbids some
   of the internally used JavaScript for security reasons, Firefox does not).

As a result, for a very long time Firefox was the browser used productively
here.

It had long been announced that new Firefox versions would not support
``legacy'' extensions anymore \[1], yet until it was finally required to
switch due to the end of security updates \[2], no larger attempt to replace
existing legacy extensions had been conducted because it had always seemed
infeasible to find adequate replacements for all extensions.

---------------------------------------------------------[ Useful Extensions ]--

Why would one need extensions in the first place? Generally, there are multiple
categories of issues addressed by the extensions which are described in the
following.

{\img{ext52}{Extensions installed in Firefox 52}}

Web Development
---------------

With modern browsers' integrated web development tools, the need for specialized
extensions has certainly decreased. Still, the following extensions were found
to useful in addition to the browser-integrated tools:

Web Developer
   Previously known as _Web Developer Toolbar_, this extension provides an
   additional menu for quickly performing tasks like resizing the browser window
   or disabling certain features. It can even be used to radically disable all
   stylesheets or images to check if pages really degrade gracefully.
   Some of the functionality is also available through browser-supplied tools,
   still _Web Developer_ allows accessing them more quickly.

User Agent Switcher
   As one would expect, this allows arbitrary user agents to be sent which is
   useful to test how pages behave differently for different browsers without
   needing the actual browser.

HttpFox
   With this extension, one could open a window similar to _Wireshark_ but
   within the browser and thus more quickly and unaffected by company policies
   prohibiting the use of Wireshark for security reasons. Of course, being
   specialized to the browser's HTTP-requests, this extension's interface was
   easier to use than Wireshark's and thus also often more quickly to navigate.

{\img{httpfox}{Opening the Ma\_Sys.ma website with HttpFox shows
the resulting browser requests.}}

User Interface Adjustments
--------------------------

For integration with MDVL user interface concepts, two addons were used to
enhance the web browser.

Classic Theme Restorer
   Mainly used to restore what one was already used to from previous Firefox
   versions but also disabling unnecessary animations, this extension was not
   strictly needed but had been successfully used to avoid unexpected
   user interface changes for some time.

{\img{classic}{The Classic Theme Restorer's preferences panel provided
fine-tuning for just about any GUI quirk changed between Firefox versions.}}

VimFx
   This enhanced Firefox with vim-like modes and keybindings and therefore
   unified the interaction with MDVL to some extent: browser, text editor and
   window manager could all be controlled with the same [H], [J], [K], [L]
   keys for navigation.

{\img{vimfx}{With VimFx being a keyboard-driven extension, there is little
GUI to show here. Here, the description of the extension's functionality with
the button and major advantage compared to other similar extensions is
highlighted by red borders.}}

Dealing with bad Websites
-------------------------

Today's Internet is full of advertising, tracking and malware. Most of it comes
in form of JavaScript (other ``active elements'' like Adobe Flash are slowly
dying). Equally annoying is the trend to use gray on white text or illegibly
small fonts. A set of extensions was used to integrate the appearance of
websites better into MDVL and disable unwanted things with the ability to
interactively override the changes if a website would not work.

NoScript
   Probably the most famous among all extensions listed on this page, NoScript
   was one of the earliest extensions added to Firefox and never again removed.
   It provides the ability to disble JavaScript by default and then activate it
   on a per-site basis, either temporarily or by updating a whitelist.
   Additionally, NoScript can block some forms of
   _Cross-Site Request Forgery_ (XSRF) attacks.

{\img{noscript}{NoScript before it was a WebExtension}}

Document Font Toggle
   After one of the more recent Firefox updates, fonts had ceased to render
   properly (as if hinting were disabled) and it seemed impossible to recover
   from that. As a radical, yet beneficial solution, all fonts were changed to
   _Terminus_. With some pages failing to work with fonts other than their own
   (especially considering the trend to use web fonts to display icons on
   pages), a means of re-enabling the pages' fonts whenever needed was realized
   by using the _Document Font Toggle_ extension. With new Firefox versions,
   font rendering improved again such that text could again be read without
   Terminus, yet the extension was kept because it still often increased
   readability and unified the font across most MDVL applications.

Monochro
   MDVL uses white-on-black text for all console windows. As file manager,
   e-mail and text editor are all console applications on MDVL, this means the
   web browser is often the only non-console application running and then it is
   also often the only application with black text on white ground by default.
   To be able to disable pages' colors and replace them by a distinguishable
   and MDVL-compatible colorset, this extension was used to be able to switch-on
   pages' colors whenever needed and otherwise display fonts and borders white
   on dark gray by default. As a side-effect this extension disables background
   images which can be useful to reduce annoying advertisements but also
   sometimes makes parts of pages' user interface illegible.

{\img{wcol}{Ma\_Sys.ma Website with Document Font Toggle and Monochro disabled,
i.e. regular rendering}}

{\img{nocol}{Ma\_Sys.ma Website with Document Font Toggle and Monochro enabled:
Fonts are forced to be Terminus and colors are removed}}

Managing Internet Information Efficiently
-----------------------------------------

The Internet is constantly changing and information is coming online and offline
at seemingly uncontrollable times. As a result, whenever important information
is found on the Internet, one should be quick to take a backup of them as not to
find them deleted if the same information is needed later. For managing online
sources in scientific (and other) writing, it is also beneficial to copy pages
not only by their link but also backup their content.

Four extensions were used to provide for most of the cases:

Saved From URL
   This plugin adds the original URL a page was downloaded from to all pages
   saved through the browsers ``Save As'' dialog also available through
   [CTRL]-[S]. When managing online sources of information, it is advisable to
   save all used pages to a directory with [CTRL]-[S]. By default, this will
   always include the page title and author because those are commonly displayed
   on the page itself but interestingly, the URL is rarely contained in pages.
   This extension adds a HTML comment with the download URL to every page
   downloaded and thus eliminates the need for managing URLs externally when
   gathering information from online sources.

Fire Captor
   This extension allows screenshots of whole pages to be taken (without
   limitation by screen height or width). Today, the browser-integrated
   developer tools provide most of this extension's functionality and external
   programs like `wkhtmltopdf` are often better because they produce a
   PDF output rather than a bitmap. As a result, this addon was generally
   disabled but could be enabled whenever special circumstances required
   screenshots of pages to be taken from within the browser.

Print Edit
   This allows portions of pages to be removed before printing which includes
   the ability to export parts of webpages to PDF which is more difficult
   and sometimes impossible by other means.

ScrapBook
   This extension manages a directory of downloaded pages and additionally
   allows pages to be downloaded recursively. While different external tools
   (often also with higher performance) exist to achieve this task, e.\,g.
   `wget` and `httrack`, those generally do not support all the features of
   modern browsers and thus fail to download more complex pages. ScrapBook
   always achieved the most complete recursive download of websites and could
   be operated quickly because it automatically managed the download location
   of pages. Unlike with [CTRL]-[S], a single downloaded page includes the
   download URL by default and comes as a single folder instead of a file and
   separate folder. This way, renaming the directory is less an issue than
   with [CTRL]-[S]. With improvements in D5Man, ScrapBook's importance should
   decrease but as of now, the extension is still required.

{\img{scrapbook}{The power of ScrapBook is hard to capture in a screenshot but
the important feature of recursively saving pages can be seen in the
``Save Page As...'' dialog}}

Extensions used before Firefox 60
---------------------------------

The following table provides an overview of the extensions used before along
with a number of `#`-characters to indicate their importance (more `#` means
more important).

	oooooooooooooooooooooooooooooooooooooo
	  Extension               Importance
	++++++++++++++++++++++++++++++++++++++
	  WebDeveloper            ###
	  User Agent Switcher     ##
	  HttpFox                 ######
	  Classic Theme Restorer  #
	  VimFx                   ##########
	  NoScript                ##########
	  Document Font Toggle    ####
	  Monochro                ######
	  Saved From URL          ##########
	  Fire Captor             #
	  Print Edit              #
	  ScrapBook               #########
	oooooooooooooooooooooooooooooooooooooo

As one can see, the most important extensions were _VimFx_, _NoScript_ and
_Saved From URL_ with _ScrapBook_ still being very important.

--------------------------------------------------[ Making Firefox 60 Usable ]--

Upgrading to Firefox 60 resulted in all of the extensions being deactivated
except for _Web Developer_, _NoScript_. At least the _User Agent Switcher_ was
also available for Firefox 60 and could thus be installed again.

With many of the extensions initially lost, functionality was restored by
searching for compatible extensions with similar behaviour than the old ones and
by setting aside the less frequently used ones.

The following extensions were considered unimportant enough to not need a
replacement: _Classic Theme Restorer_, _Fire Captor_, _Print Edit_ (all with
importance `#`).

As a result, replacements for the following extensions needed to be sought

 * HttpFox
 * VimFx
 * Document Font Toggle
 * Monochro
 * Saved From URL
 * ScrapBook

Extensions without sensible Replacements
----------------------------------------

HttpFox
   This extension could not be replaced by any other extension.
   For web development, that it is acceptable because Wireshark and the
   browser-integrated tools provide the needed functionality, yet not having the
   flexibility of a Wireshark-like tool usable in restricted environments is a
   major drawback.

Saved From URL
   Very sadly, there does not seem to exist a replacement for this simple
   functionality. It might be possible that such an extension is written
   in the future given its clearly specified and single functionality.
   If anybody knows a solution/replacement for this, please notify
   `Ma_Sys.ma@web.de`.

Replacement Extensions for VimFx and ScrapBook
----------------------------------------------

Vimium(https://github.com/philc/vimium)
   The description of VimFx already lists some alternatives and _Vimium_ is
   a really good replacement. The only disadvantage compared to before seems
   that it does not focus newly created tabs by default.

Web\ ScrapBook(https://github.com/danny0838/webscrapbook)
   While it is similar to _ScrapBook_ by its name, _Web ScrapBook_ only provides
   a fraction of the useful features from _ScrapBook_: It does not currently
   seem to support recursive downloads and a sidebar/menu with a list of all
   downloaded pages is also missing. Still, it stores the original URL and
   is better than [CTRL]-[S] as explained for _ScrapBook_ before.
   Time will tell how well it replaces _ScrapBook_ when used regularly.

The Special Cases: Document Font Toggle and Monochro
----------------------------------------------------

Although some extensions exist for realizing similar functionality to
_Document Font Toggle_ and _Monochro_, all of these suffer from some drawbacks.

Quite a few extensions for decolorization exist, but the closest alternative
to _Monochro_ did not disable the pages' colors by default rendering it useless
to prevent the ``readiation'' of white from a large browser window when opening
a new page.

To make fonts and colors work as expected, it seemed apt to realize this with an
userstyle addon which would be able to toggle the application of a stylesheet.
In its most simple form, the stylesheet could look as follows:

{\code{css}}

	/* Userstyle concept CSS for Terminus and only two colors */
	* { font-family: Terminus; background-color: #3c3c3c; color: #ffffff; }

In practice, some more declarations were needed but even with `!important`
added to all CSS, userstyle extensions' CSS would still be superseded by pages
with `!important` in their CSS. In the end, a solution _without extensions_
proved to be closest to the previous behavior.

Using the file `<profile>/chrome/userContent.css`, Firefox users can supply
a stylesheet which overrides everything -- it can even change Firefox' GUI!
With the following stylesheet, an adequate implementation of ``Terminus and
few colors only'' can be realized as follows:

	/* Ma_Sys.ma User Style 1.0.0, Copyright (c) 2018 Ma_Sys.ma */
	/* For further info send an e-mail to Ma_Sys.ma@web.de */
	
	/**/
	
	@-moz-document regexp("^(http|ftp|file).*$") {
		* {
		    color: white !important;
		    background-color: #3c3c3c !important;
		    border-color: white !important;
		    outline-color: white !important;
		    opacity: 1.0 !important;
		    font-family: Terminus !important;
		    text-shadow: none !important;
		    box-shadow: none !important;
		}
		a, button, input[type="submit"] {
		    color: #00cccc !important;
		    border-color: #00cccc !important;
		}
		body, div, p {
		    background-image: none !important;
		}
	}

It might look a bit more complex than needed, but there are quite a few annoying
visual CSS properties (shadows, opacity, etc.) which need to be considered.
Links and buttons should appear as clickable elements (realized by a different
color here) and full-page background images should be disabled without
disabling the smaller images used for modern webpages' GUIs.

The `@-moz-document regexp` disables the stylesheet for other elements than
pages' contents i.\,e. the GUIs of WebExtensions are not formatted (this is
most important for uMatrix to work properly).

This solution is considered OK at the moment, but has the major drawback of
being difficult to disable temporarily: Pressing [F12] one can open the
developer tools. Then one can activate _Browser styles_ under
Inspector >> Computed and go to the _Style Editor_ and find `userContent.css`.
There, one removes the last `/` to comment-out the whole file which makes
pages appear using only their styles.

{\img{userstyle}{Deactivating the userContent.css is not as simple as pressing
the Monochro button was in the past.}}

uMatrix
-------

With many things changing due to the upgrade, an alternative to _NoScript_ was
also tested and found a good replacement (although NoScript is available
for Firefox 60 onwards): uMatrix(https://github.com/gorhill/uMatrix). This
extension gives better control about what exactly is blocked and uses rules
rather than a whitelist to flexibly allow portions of active elements for pages.

It is also slightly more complex to use but the user interface is easy enough to
learn while using the extension :) .

Extensions installed for Firefox 60
-----------------------------------

Finally, here is an overview on the changes and how well they went.

    oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
      Previous                 Now                  Impact Summary
    ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
      Web Developer            Web Developer        + OK
      User Agent Switcher      User Agent Switcher  + OK
      HttpFox                  ~                    - Bad if no Wireshark
      Classic Theme Restorer   ~                    + OK
      VimFx                    Vimium               + OK
      NoScript                 uMatrix              + better
      Document Font Toggle     `userContent.css`    - Switching it off is
      Monochro                 `userContent.css`    - much more complicated
      Saved From URL           ~                    - BAD :(
      Fire Captor              ~                    + OK
      Print Edit               ~                    + OK
      ScrapBook                Web ScrapBook        - lacks important features
    oooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

The probably largest negative impact is in the
_Managing Internet Information Efficiently_ category: The advanced downloading
abilities of _ScrapBook_ and the simplicity and workflow efficiency of
_Saved From URL_ are lost.

Additionally, there is some setback on the MDVL GUI integration:
`userContent.css` is a partial and difficult-to-handle replacement for the
handy extensions used before.

{\img{ext60}{Extensions installed in Firefox 60}}

All in all, the number of extensions installed was reduced dramatically which
is a clear advantage from a security point of view.

-------------------------------------------------------------------[ Summary ]--

Why would this whole thing interest you? This post serves at least these three
functions:

 1. It is an opinion that there is some real friction and
    disadvantages of giving up compatibility with legacy extensions,
 2. By describing the extensions and their functionality, anybody interested
    might get some ideas about things one can do with a flexible web browser
    such as Firefox and it might inspire reader's to try out some of the
    extensions or think about the underlying issues -- integration into a local
    working environment and persisting (in?) the shortlived web.
 3. This is also some form of ``internal'' documentation which can be used to
    restore browser behaviour and benchmark new browsers to see if they support
    the special functionality realized with the extensions here.

To summarize, it might seem that about 10 hours of work just because of a
security upgrade is quite a bad thing. And although some things remain rough
because no adequate replacements exist for some extensions, Firefox is back to
a usable state and with everything being WebExtensions, the door to switching
to another compatible browser (e.\,g. Chromium) in the future is no longer
sealed.

Still, with all the interesting and worrying development going on for web
technology, it is not clear if the great compatibility between Firefox and
Google Chrome is not probably a dangerous thing because it gives up some of the
great flexibility previously uniquely found in Firefox extensions. This could
strengthen Google Chrome in the long run because there is technically less
advantage of Firefox over Chrome and one can doubt that Mozilla can keep up with
Google in terms of rendering engine performance. This in turn could result in a
monopoly for Google Chrome.

----------------------------------------------------------------[ References ]--

\[1] Caitlin Neiman: _Timeline for disabling legacy add-ons on
addons.mozilla.org | Mozilla Add-ons Blog_, url(https://blog.mozilla.org/
addons/2018/08/21/timeline-for-disabling-legacy-firefox-add-ons/),
checked 2018/09/09.

\[2] Moritz Muehlenhoff: _Debian -- Security Information -- DSA-4287-1
firefox-esr_, url(https://www.debian.org/security/2018/dsa-4287),
checked 2018/09/09.


Zum Seitenanfang