- CachingNewsServersForDebian

Introduction

NNTPCache has been removed from Debian due to licensing restrictions. The version in Debian etch could be used in Debian lenny, but recently I upgraded to Debian squeeze and switched from 32-bit to 64-bit and this ancient version of NNTPCache would no longer work.

This article describes recent experiences trying to obtain and build a replacement, mainly in order to record the various error messages I encountered, in the hope that it will help other people.

If you're just interested in how to get it working then skip down to building Christoph Meerwald's version of NewsCache from source revision 52 or building Christoph Meerwald's version of NewsCache into Debian package revision 52; many thanks to Christof Meerwald for speedy feedback and updates!

Overview of what I tried

I tried to build a 64-bit Debian package of NNTPCache, based on the last-available sources and the corresponding .dsc and .diff.gz files from Debian. The intention was to make a 64-bit version and to "iron out" any dependency issues by building on the target platform. The procedure I used, which was not successful, is below.

I looked for similar products using apt-cache and Google.

I tried Herbert Straub's version of NewsCache. He provides 32-bit Debian packages but I wanted to build it for Debian in order to "iron out" any dependency issues. The procedure I used, which was not successful, is below.

I tried to compile Herbert's version from sources. This was not successful. The procedure I used is below.

I could not try to install Herbert's Debian packages because they are 32-bit and my target system is 64-bit.

Christof Meerwald maintains a fork of NewsCache. This version was updated more recently and has fewer dependencies than Herbert's version.

Before mid-April 2011, I tried to compile Christoph's version (subversion revision 38) from sources. Initially, this was not successful. The procedure I used is below.

Christoph provides 64-bit Debian packages for Ubuntu, but I wanted to build it for Debian in order to "iron out" any dependency issues. Initially, this was not successful, producing the same errors as when I tried to compile from sources.

I also tried Christoph's 64-bit Debian packages for Ubuntu, with the same result.

Christof then did some updates (run svn log http://svn.cmeerw.net/newscache/ for details).

At the end of April 2011, I tried again to compile Christoph's version (subversion revision 52) from sources. This worked! The procedure I used is below.

I also tried again to build Christoph's version for Debian in order to iron out any dependency issues. This worked! The procedure I used is below.

I also tried Christoph's Christoph's 64-bit Debian packages for Ubuntu. This worked!

Procedure: building NNTPCache

1. Create a new directory to work in by running:

   mkdir nntpcache-build && cd nntpcache-build 

2. Download NNTPCache, check its integrity and rename it to the name that the Debian build tools will want by running:

   wget http://iq.org/~proff/nntpcache.org/pub/nntpcache/nntpcache-3.0.1.tar.gz
   echo "26f142bf39d76699d2b9620c5940a4dd  nntpcache-3.0.1.tar.gz" | md5sum -c
   mv nntpcache-3.0.1.tar.gz nntpcache_3.0.1.orig.tar.gz 

3. Download the files necessary to patch, compile and package NNTPCache from this site by running:

   wget -O nntpcache_3.0.1-4.dsc 'http://dione.no-ip.org/AlexisWiki/BuildingNntpCacheForDebian?action=AttachFile&do=get&target=nntpcache_3.0.1-4.dsc'
   wget -O nntpcache_3.0.1-4.diff.gz 'http://dione.no-ip.org/AlexisWiki/BuildingNntpCacheForDebian?action=AttachFile&do=get&target=nntpcache_3.0.1-4.diff.gz' 

4. Create the package by running:

   dpkg-source --no-check -x nntpcache_3.0.1-4.dsc
   cd nntpcache-3.0.1
   dpkg-buildpackage -us -uc -rfakeroot
   cd .. 

5. Check the contents of the debs you've produced by running:

   dpkg --contents nntpcache_3.0.1-4_amd64.deb
   dpkg --contents nntpcache-dev-doc_3.0.1-4_all.deb 

6. Install the nntpcache deb with:

   dpkg -i nntpcache_3.0.1-4_amd64.deb 

7. Modify the configuration files /etc/nntpcache.* according to local requirements.

8. Stop and start the NNTP server as follows:

   service nntpcache restart 

9. Wait a few moments and then try to manually connect to the NNTP server by running:

   telnet ($uname -n) nntp                #   don't use 'localhost' unless you granted access to localhost 

10. This failed with the error:

    spaghetti# telnet spaghetti nntp
    Trying 192.168.1.13...
    Connected to spaghetti.pasta.net.
    Escape character is '^]'.
    503 NNTPCache-3.0.1 Failing server rebuild in progress (0 groups complete, at least 3000 groups to go. Please try again later. If you think this is an error, get your news admin to check minGroups (in nntpcache.config), nntpcache.servers and the nntpcache web-server output
    Connection closed by foreign host.
    spaghetti# 

    This message does not necessarily indicate an error. But I then ran:

    cat /var/log/news/news.err 
    Apr  9 13:05:00 spaghetti nntpcache-expire[14219]: expire.c:228: usedblocks = 57.6%, usedinodes = 29.1%, maxArtAge = 46s, expire ended
    Apr  9 13:05:00 spaghetti nntpcache-update[14224]: nntpcache.c:290: page error 

    and an examination of the code revealed that this was caused by a SIGSEGV error, which looks very much like the error described at http://groups.google.com/group/news.software.nntp/msg/b18c6c5558268422.

Procedure: building NNTPCache from source

1. Run:

   wget http://iq.org/~proff/nntpcache.org/pub/nntpcache/nntpcache-3.0.1.tar.gz
   tar xzf nntpcache-3.0.1.tar.gz
   cd nntpcache-3.0.1
   ./configure --prefix=/usr/local/opt/nntpcache-3.0.1
   make
   make install 

2. Modify the configuration files in /usr/local/opt/nntpcache-3.0.1 according to local requirements.

3. Run:

   /usr/local/opt/nntpcache-3.0.1/sbin/nntpcached 

4. After a couple of minutes I tried to manually connect to the NNTP server by running:

   telnet ($uname -n) nntp 

   but this failed, exactly as described in the above section 'Procedure: building NNTPCache'.

Procedure: building Herbert Straub's version of NewsCache

1. Create a new directory to work in by running:

   mkdir newscache-build && cd newscache-build 

2. Download NewsCache and the files necessary to patch, compile and package it and check their integrity:

   wget http://src.linuxhacker.at/packages/debian/sid/socket++_1.12.10.orig.tar.gz
   wget http://src.linuxhacker.at/packages/debian/sid/socket++_1.12.10-1.dsc
   wget http://src.linuxhacker.at/packages/debian/sid/socket++_1.12.10-1.diff.gz
   wget http://src.linuxhacker.at/packages/debian/sid/newscache_1.1.12.orig.tar.gz
   wget http://src.linuxhacker.at/packages/debian/sid/newscache_1.1.12-1.dsc
   wget http://src.linuxhacker.at/packages/debian/sid/newscache_1.1.12-1.diff.gz 

3. Install prerequisite software by running:

   apt-get install libtool 

4. Unpack the source code by running:

   dpkg-source --no-check -x  socket++_1.12.10-1.dsc
   dpkg-source --no-check -x  newscache_1.1.12-1.dsc 

5. Patch the code to compile with Debian squeeze's version of g++ as follows:

   1. Edit socket++-1.12.10/socket++/sig.cpp and change lines 52, 93, 123, 136 and 150 to include casts. This means changing:

      ... sighnd_type ... 

      to:

      ... (void(*)(int)) sighnd_type ... 

   2. Edit newscache-1.1.12/NServer-0.9.1/sstream.cc and change line 33 from:

                 << (unsigned int) this << "\n"); 

      to:

                 << (char *) this << "\n"); 

   3. Edit newscache-1.1.12/NServer-0.9.1/readline.cc and newscache-1.1.12/test/tlock.cc and add, near the top with the other #include lines:

      #include <stdlib.h> 

6. Create the packages by running:

   cd socket++-1.12.10
   dpkg-buildpackage -us -uc -rfakeroot
   cd ..
   dpkg -i socket++1_1.12.10-1_amd64.deb socket++-dev_1.12.10-1_amd64.deb
   cd newscache-1.1.12
   dpkg-buildpackage -us -uc -rfakeroot
   cd ..
   dpkg -i newscache_1.1.12-1_amd64.deb 

7. Copy /etc/newscache.conf-dist to /etc/newscache.conf and modify according to local requirements.

8. Run:

   mkdir -p /var/cache/newscache
   chmod 777 /var/cache/newscache 

9. Start the NNTP server as follows:

   service newscache start 

10. I then did a test posting to de.test, which hung. Strace revealed:

    12625 write(1, "340 send article to be posted.\r\n", 32) = 32
    12625 read(1, "Newsgroups: de.test\r\nSubject: aaaaaaaaaaaa\r\nFrom: Alexis Huxley <ahuxley@xxx.xxx>\r\nUser-Agent: nn/6.7.3\r\n\r\naaaaaaaaaaaaaaaa\r\n.\r\n", 8192) = 128 

    and top showed:

    12625 news      20   0 27224 3556 3176 R  100  1.4   2:04.78 newscache 

    which suggested the program was stuck in a tight infinite loop.

Procedure: building Herbert Straub's version of NewsCache from source

1. Install libsocket++ as described above.

2. Run:

   wget http://src.linuxhacker.at/NewsCache/NewsCache-1.2rc6.tar.gz
   tar xzf NewsCache-1.2rc6.tar.gz
   cd NewsCache-1.2rc6  

   (Note that the web page references a patch, but the link is broken.)
3. Patch tlock.cc and readline.cc as above.

4. Run:

   ./autogen
   ./configure --prefix=/usr/local/opt/newscache-1.2rc6 

5. Modify config.h and change:

   #define SYSCONFDIR "${prefix}/etc" 

   to agree with the prefix you used above; e.g.

   #define SYSCONFDIR "/usr/local/opt/newscache-1.2rc6/etc" 

   (This must have been the effect of that patch above.)

6. Run:

   make
   make install 

7. Copy /usr/local/opt/newscache-1.2rc6/etc/newscache.conf-dist to /usr/local/opt/newscache-1.2rc6/etc/newscache.conf and modify according to local requirements.

8. Run:

   mkdir -p /var/cache/newscache
   chmod 777 /var/cache/newscache 

9. Run:

   /usr/local/opt/newscache-1.2rc6/sbin/newscache 

10. I then did a test posting to de.test. The first attempt succeeded but all later attempts failed, exactly as described in the above section 'Procedure: building Herbert Straub's version of NewsCache'.

Procedure: building Christoph Meerwald's version of NewsCache from source revision 38

1. Run:

   REVISION=38
   svn co -r $REVISION http://svn.cmeerw.net/newscache/trunk newscache-trunk
   PREFIX=/usr/local/opt/newscache-svn$REVISION
   cd newscache-trunk
   ./autogen
   ./configure --prefix=/usr/local/opt/newscache-svn$REVISION 

2. Modify SYSCONFDIR in config.h by running:

   perl -pi -e "s@^#define SYSCONFDIR.*@#define SYSCONFDIR \"$PREFIX/etc\"@" config.h 

   (Regarding the need for this, see 'Procedure: building Herbert Straub's version of NewsCache from source' below.)

3. Run:

   make
   make install 

4. Copy $PREFIX/etc/newscache.conf-dist to $PREFIX/etc/newscache.conf and modify according to local requirements.

5. Run:

   mkdir -p /var/cache/newscache
   chmod 777 /var/cache/newscache 

6. Run:

   $PREFIX/sbin/newscache 

7. I then did a test posting to de.test. The first attempt succeeded but all later attempts failed. The client received:

   Remote error: 340 send article to be posted. 

   and NewsCache logged:

   Apr 10 11:04:30 spaghetti NewsCache[5634]: nnrpd: access_entry name matched:
   Apr 10 11:04:30 spaghetti NewsCache[5634]: fiori.pasta.net [192.168.1.6] connect
   Apr 10 11:04:30 spaghetti NewsCache[5634]: NServer::NServer() hostname set to: spaghetti.pasta.net
   Apr 10 11:04:30 spaghetti NewsCache[5634]: fiori.pasta.net [192.168.1.6] MODE READER
   Apr 10 11:04:30 spaghetti NewsCache[5634]: fiori.pasta.net [192.168.1.6] LIST
   Apr 10 11:04:30 spaghetti NewsCache[5634]: fiori.pasta.net [192.168.1.6] LIST active.times
   Apr 10 11:04:39 spaghetti NewsCache[5634]: fiori.pasta.net [192.168.1.6] POST
   Apr 10 11:04:39 spaghetti NewsCache[4981]: 5634 caught signal 11
   Apr 10 11:04:39 spaghetti NewsCache[5649]: nnrpd: access_entry name matched:
   Apr 10 11:04:39 spaghetti NewsCache[5649]: fiori.pasta.net [192.168.1.6] connect
   Apr 10 11:04:39 spaghetti NewsCache[5649]: NServer::NServer() hostname set to: spaghetti.pasta.net
   Apr 10 11:04:39 spaghetti NewsCache[5649]: fiori.pasta.net [192.168.1.6] MODE READER
   Apr 10 11:04:39 spaghetti NewsCache[5649]: fiori.pasta.net [192.168.1.6] POST
   Apr 10 11:04:43 spaghetti NewsCache[5649]: fiori.pasta.net [192.168.1.6] failed POST
   Apr 10 11:04:43 spaghetti NewsCache[5649]: NewsCache.cc input stream eof!
   Apr 10 11:04:43 spaghetti NewsCache[5649]: fiori.pasta.net [192.168.1.6] exit articles 0 groups 0 

   and a simultaneous strace of the process showed:

   spaghetti# strace -f -p 4981
   ...
   5634  alarm(600)                        = 0
   5634  recvfrom(1, "POST\r\n", 4096, 0, NULL, NULL) = 6
   5634  alarm(0)                          = 590
   5634  gettimeofday({1302426279, 796117}, NULL) = 0
   5634  sendto(4, "<62>Apr 10 11:04:39 NewsCache[56"..., 71, MSG_NOSIGNAL, NULL, 0) = 71
   5634  sendto(1, "340 send article to be posted.\r\n", 32, 0, NULL, 0) = 32
   5634  recvfrom(1, "Newsgroups: de.test\r\nSubject: ee"..., 4096, 0, NULL, NULL) = 133
   5634  --- SIGSEGV (Segmentation fault) @ 0 (0) ---
   4981  <... accept resumed> 0, NULL)     = ? ERESTARTSYS (To be restarted)
   4981  --- SIGCHLD (Child exited) @ 0 (0) ---
   ... 

   I examined NewsCache.cc's ns_post() function and decided to comment out two blocks of code, one at a time. Firstly I commented out the block of code dealing with client authentication:

                   // check whether client is authorized to post this article.
                   string newsgroups = art.getfield("newsgroups:");
                   string::size_type i1 = 0, i2 = newsgroups.find(",");
                   for (;;) {
                           string group = newsgroups.substr(i1, i2);
                           if (clt->access_entry->postTo.
                               matches(group.c_str()) <= 0) {
                                   (*clt->
                                    co) << "480 authentication required\r\n";
                                   return 0;
                           }
                           if (i2 == string::npos)
                                   break;
                           i1 = i2 + 1;
                           i2 = newsgroups.find(",", i1);
                   } 

   Commenting this out made no difference. Secondly I commented out the block of code dealing with actually posting the article:

                   // client is authorized to post the article
                   clt->srvr->post(&art);
                   (*clt->co) << "240 Article posted\r\n"; 

   Commenting this out made NewsCache able to continue without error - although obviously it had not posted the article submitted by the client. I know zero C++ and a deeper examination was not possible.

Procedure: building Christoph Meerwald's version of NewsCache from source revision 52

1. Run:

   REVISION=52
   svn co -r $REVISION http://svn.cmeerw.net/newscache/trunk newscache-trunk
   PREFIX=/usr/local/opt/newscache-svn$REVISION
   cd newscache-trunk
   ./autogen
   ./configure --prefix=/usr/local/opt/newscache-svn$REVISION 

2. Run:

   make
   make install 

3. Copy $PREFIX/etc/newscache.conf-dist to $PREFIX/etc/newscache.conf and modify according to local requirements.

4. Run:

   mkdir -p /var/cache/newscache
   chmod 777 /var/cache/newscache 

5. Run:

   $PREFIX/sbin/newscache 

Procedure: building Christoph Meerwald's version of NewsCache into Debian package revision 52

1. Create a new directory to work in by running:

   mkdir newscache-build && cd newscache-build 

2. Download NewsCache and the files necessary to patch, compile and package it:

   wget http://download.cmeerw.net/debian/newscache/source/newscache_1.3.ipv6.orig.tar.gz
   wget http://download.cmeerw.net/debian/newscache/source/newscache_1.3.ipv6-1~cmeerw.11.dsc
   wget http://download.cmeerw.net/debian/newscache/source/newscache_1.3.ipv6-1~cmeerw.11.diff.gz 

3. Install prerequisite software for building NewsCache:

   apt-get install libgnutls-dev libpam0g-dev 

4. Create the package by running:

   dpkg-source --no-check -x *.dsc
   cd newscache-1.3.ipv6
   dpkg-buildpackage -us -uc -rfakeroot
   cd .. 

5. Check the contents of the debs you've produced by running:

   dpkg --contents newscache_1.3.ipv6-1~cmeerw.11_amd64.deb 

6. Remove the prerequisite software for building NewsCache:

   apt-get --purge remove libgnutls-dev libpam0g-dev 

   and install the ones needed to run NewsCache:

   apt-get install libgnutls26 libpam0g 

7. Install the NewsCache deb with:

   dpkg -i newscache_1.3.ipv6-0cmeerw.9_amd64.deb 

8. Copy /etc/newscache.conf-dist to /etc/newscache.conf and modify according to local requirements.

9. Start the NNTP server as follows:

   service newscache start 

See also

• http://cmeerw.org/debian/ (Christoph Meerwald's fork of NewsCache)

• http://www.linuxhacker.at/NewsCache/download (Herbert Straub's original NewsCache)

• http://iq.org/~proff/nntpcache.org/ (NNTPCache home page)

• http://laurentparenteau.com/blog/2010/08/building-sourcebinary-deb/ (short description of building debs from sources)

• http://ftp.cica.es/Guadalinex/repositorio/pool/non-free/n/nntpcache/ (another place to get the NNTPCache sources - was accessible April 2011 but maybe not much longer)

• http://groups.google.com/group/news.software.nntp/msg/b18c6c5558268422

• Computing

• AlexisHuxley

---

CategoryArticle