Multi-threaded NewsWatcher


The need for speed...

Why does it take MT-NewsWatcher so long to download headers?

The original NewsWatcher fetches only Subject and Author headers from the news server, and does so using the XHRD command which must be given once for each header type. MT-NewsWatcher can obtain much more information about articles, such as their posting date and line count, so making decisions about which articles to read easier for the user. However, this incurs an overhead in terms of the time taken to download such information.

You can control whether MT-NewsWatcher needs to download header in addition to the Subject header (which is always downloaded) using the settings in the 'Subject window' panel of the preferences. MT-NewsWatcher implements a new way of getting header information from the server, using the 'XOVER' command. When this is activated, fetching headers is usually much faster.

However, as explained below, downloading time is not a simple function of these settings. This document explains how the various settings impact downloading time.

What do those server options mean? In the 'Server options' panel of the Preferences dialog, there are a number of setting that determine how MT-NewsWatcher communicates with your news server. Most of them have speed implications.

Use XPAT command for searches
Searching article headers for a certain string can be performed either by the news server, or by MT-NewsWatcher itself. If this option is checked, searches are performed by sending an 'XPAT' request to the server, which then responds with a list of matching articles. Although this may take some time, the job of searching is done by the server. If this option is unchecked, MT-NewsWatcher must download all the headers for that group and search them itself, which is much slower and requires more memory.

Nearly all servers support XPAT, so you should turn this option on, and only uncheck it if the server sends back an error message when you try to search.

Use XOVER command to get headers
This is the main option that affects speed. When active, article header information is downloaded in larger chunks, which is much faster. It is described in more detail below.

Again, most servers support XOVER, so you should turn this on. Beware that some servers may cause a warning message to be displayed the first time you use it; the reason for this is described below.

Don't open new connection when checking for new articles
Some servers return bogus article counts if you reuse an existing connection to request group article information. In this case, the client has to open a new connection to check for new articles.

This is not a problem with most servers, so this option should be checked. If the 'Check for new articles' command never returns new articles, you could try turning this option off.

Don't send MODE READER command
Some servers require that news clients identify themselves as such, by sending the 'mode reader' command (to distinguish them from other servers with which your server exchanges news). Most don't, so you should turn this option on.

Allow up to N simultaneous connections
This controls the maximum number of concurrent connections to the news server that can be opened. 4 is a sensible value. More detail is given in the threading document.
What determines which headers are downloaded? There are four settings that determine which headers need to be obtained from the server. These are:
'Subject window' preferences
The three checkboxes in this preferences panel control whether or not authors, dates and line counts are shown in subject windows. If checked, this information obviously needs to be obtained from the server.

'Server options' preferences
The 'Use XOVER command to get article headers' is the important setting here; its workings are described below. If checked, download time will not be affected by the choice of options in the 'Subject window' preferences.

'Thread using references' option
This option, in the 'Newsreading options' panel, specifies that MT-NW should attempt to build discussion threads based on the contents of the References header. When active, Message-ID and References headers are required from the server, because they are needed to build the threads. Both these headers are included in XOVER responses.
The set of filters that apply to the group being downloaded may also require that additional information be obtained from the server. Extra headers will need to be downloaded if:

The filter acts on a header that is not one of those being displayed, as controlled by the 'Subject window' settings.
The filter acts on a header that is not contained in the XOVER data.
Of course, if more than one filter acts on an additional header, that header only needs to be downloaded once.

In summary, all headers which are either to be displayed, or filtered, or used to build threads are required from the server. If XOVER is active, then most information will be contained in the response to the XOVER command, and additional headers will only be required if they are not contained in the XOVER data, which is described below.
The XOVER command When the news client (e.g. MT-NewsWatcher) requests XOVER information for a set of articles, the server replies with a block of data for each article that must contain the following headers:

Date MT-NW converts this into your preferred date format
Message-ID Needed for reference-based threading
References Needed for reference-based threading
Byte count Unused
Line count

and which may (depending on your server type) contain this header:

Xref Contains cross-posting information

Because this information is sent in big chunks, this is much faster than fetching each of these headers independently. When receiving these data, MT-NW processes them, discards information that is not required either for display or filtering, and keeps the rest.

With XOVER active, almost all the information that you will normally need for display, threading, and a basic set of filters, will be downloaded in one go, which is fast. The exceptions are some older news servers which don't include Xref information (which is useful for filters designed to catch excessive cross-posting), and filters using the less common headers like X-Mailer, and X-Newsreader.

Why do I a get a warning with XOVER? When you activate XOVER for the first time, you may see a warning which says:

This news server fails to respond to a command which provides information about the format of XOVER request results.

and gives you the options of either continuing at your peril, or cancelling.

This occurs because your news server does not conform to some recent specifications about the format of XOVER responses. News clients are supposed to be able to get information from the server on the content of XOVER responses by using the LIST OVERVIEW.FMT command. MT-NewsWatcher uses this to determine what to expect from an XOVER command. However, some servers (notably those by Microsoft and the DNEWS server) don't understand that LIST command. In these cases, MT-NewsWatcher assumes that such responses must contain the headers listed in the table above, and if it finds Xref headers, it will use these as well. But to warn you that it is making these assumptions (which might, for some future server, fail), you will see this warning dialog. Hit the 'Continue' button to continue to use XOVER with the assumed format.

Refiltering MT-NewsWatcher stores those headers that it has downloaded for a group until you close the subject window. If you edit a filter and then re-filter the articles, new headers will only be downloaded if you have created a new filter that uses a header not already obtained for this group.


Download Basics Speech Recognition Filtering Cool Features