Home Uniquely NZ Travel Howto Pauline Small Firms Search
Ubuntu on the Lookout
Browsing, Email, Contacts, Calendars, Tasks and SMS

Contents of this page

Introduction

Lookout - noun, a vantage point with good communications commanding a wide view, from which a watch is kept on the surroundings.

This page covers the main Ubuntu programs which interact with the outside world and are largely dependent on the Internet, namely Browsers, Email, Contacts, Calendars and Tasks. It is one of a group of specialised pages which have been extracted from and form part of Fun with Ubuntu Linux.

Browsers

Firefox

Firefox is generally accepted as being the best and safest browser available - it is Open Source and available for Windows, Mac and Linux. There are many 'extensions' available which can be easily loaded to extend its functionality if required. I will mention a few that I use below. Firefox is capable of importing almost anything useful from Internet Explorer including Favourites, Cookies and History so the transition under Windows is very easy. Transfers to Linux is equally easy.

Firefox does not need to be installed in Ubuntu as it is the standard browser in all versions of Ubuntu. Hardy Heron and latter have Firefox 3.0 installed and by Lucid and Maverick Firefox 3.6 was standard that is the version I will be considering initially along with the changes in Firefox 5.0 installed with Natty 11.04. There is an extension to Firefox automatically installed which integrates it into the Repository system so some extensions are available from the repositories and, along with the main program, are automatically updated but see below as this extension is somewhat of a poisoned chalice and can cause more problems than it solves. I usually do a little configuration to make it easier to use when I install it for friends or on my clients machines.

Configuring Firefox

Firefox extensions

Firefox, like all the Mozilla programs has a built in facility to run small programs called 'extensions' to customise and add extra facilities and functionality. There are now hundreds, if not thousands, of extensions available which are added by an internal mechanism and most work under Windows, Linux and Mac operating systems. As well as those you add yourself you may find that other programs such as Windows Virus checkers (AVG is an example) add them to enhance there own facilities - these may not be compatible between operating systems. The reason I mention this will become apparent soon. I have found some of the following to be essential or useful to the way I work. Extensions in Firefox and other are usually added by: Tools -> Add-ons -> Get Add-ons -> Browse all Add-ons which takes you to a web page where you can search for an add-on that does what you require and download it as an .xpi file to your desktop.

Googlebar (or Google Toolbar)

I have been using Googlebar for a long time and still prefer it to the official Google Toolbar - one or other is an essential addition if you do a lot of searching. Googlebar adds an icon to the main toolbar customisation so you can hide and display it when you need to have a larger viewing area.

FireFTP

An easy way to gain two panel FTP functionality without installing a separate program

User Agent Switcher

There are still web sites that check the browser and refuse to display pages in browsers other than Internet Explorer. This add-on enables Firefox to 'pretend' to be IE, Netscape or Opera and allow most pages to display.

Caught-up (ITV Player)

Not available from the Firefox extensions site due to problems with advertisers as it enable adverts to be bypassed. Still available from http://chris.orr.me.uk/caughtup/. I do not use it but I found and tried it out for a friend who is very impressed with it.

Install the latest Firefox from the mozilla firefox-stable PPA

Natty uses the latest Firefox 5 and this is now available from the mozilla teams firefox-stable PPA. Note you should not use Ubuntuzilla PPA with Natty as the versions there do not integrate with the Unity Desktop or AppArmor and you should remove the repository before upgrading.

The thunderbird-stable PPA which can be installed in Lucid and Natty by the following sequence of commands in a terminal:

sudo add-apt-repository ppa:mozillateam/firefox-stable
sudo apt-get update
sudo apt-get upgrade

This version is significantly improved but you may need to reconfigure your toolbars as items such as the + for a new tab may have gone. It is easiest to reset the toolbars to the default then start again when you have seen the changes. I have also moved the tabs back to their usual place. You may find that some of the Extensions need to be updated and some may not yet be compatible.

 

Fix to AppArmor to use a Firefox profile on a separate drive/partition under Ubuntu Natty 11.04

I have always used a separate partition/drive for Data as well as having a separate partition for my home folder. When I upgraded to Ubuntu 11.04 I found none of my profiles would run under Classic Ubuntu or Unity. They still ran perfectly when I used the 'straight' Firefox 4 firefox-mozilla-build from the Ubuntuzilla PPA but that did not integrate with the Unity so that was not a long term solution. The error message on trying to start Firefox was:

Firefox cannot use the profile "??????" because it is in use. To continue, close the running instance of Firefox or choose a different profile.

This message is apparently given if the profile is locked or if the profile can not be found. It is clearly not a stale lock file as it also occured when I created a new profile so it would seem Firefox cannot find or has access blocked to the Profile.

I evetually narrowed the problem down even further and found my existing profiles would run in the default location ~/.mozilla/firefox or in a sub-folder below that location but nowhere else even via a symlink in ~/.mozilla/firefox. I could only create new profiles in the same locations. [ Click to reveal all the stages I then went through before finding the probable solution ]

The breakthrough came with a suggestion on AskUbuntu that I checked the AppArmor-Profile for Firefox. AppArmor ("Application Armor") is a security module for the Linux kernel designed to proactively protect the operating system and applications from external or internal threats, by enforcing good behavior and preventing application flaws, even unknown ones, from being exploited. AppArmor security policies completely define what system resources individual applications can access, and with what privileges. AppArmor works by implementing security profiles (apparmor-profiles) that restricts the capabilities of protected programs. Ubuntu has included AppArmor since 7.10 but up till now the apparmor-profile for firefox has present but not enabled.

This looked a likely cause of my problems but it took a lot of searching and background work over many days before I came up a very short and easy long term solution. [ Click to reveal all the background, detailed commands and terminal output from configuring AppArmor. ]

The bottom line is that any local modifications required to the 'policies' used by apparmor for firefox can just be added to the file /etc/apparmor.d/local/usr.bin.firefox and will remain in force through any system and firefox upgrades . In my case I want to allow access to the folder Profiles on any fixed drive, USB drive or encrypted volume from Truecrypt mounted at /media and read access to the whole of those drives

Edit the file in a terminal by:

sudo gedit /etc/apparmor.d/local/usr.bin.firefox

The resulting file in my case was:

# Site-specific additions and overrides for usr.bin.firefox.
# For more details, please see /etc/apparmor.d/local/README.

# Additions made by Peter Curtis 16 May 2011 to access profiles
# on removable media,additional filesystems and encrypted volumes
# in folder Profiles and to have read access to everything.

/media/** r,
owner /media/*/Profiles/** wmk,

Firefox has so far worked exactly as expected and required under Ubuntu 11.04 since this addition to /etc/apparmor.d/local/usr.bin.firefox whilst enjoying a high degree of protection from AppArmor

Email and Address Books

A Discussion of Email and Contacts and their transfer

Email is an important part of use of the internet, arguably second only to browsing. It is also an area where many people want to maintain information, not only in the Address Books and Contacts Lists on which people increasingly depend, but also in an audit trail of the email communications sent and received. The problem is that this information is not in a simple form like a list of documents that can be opened, it is all mixed up with various settings for the programs or in files and databases which are almost total opaque and incompatible between email programs such as Outlook Express, Outlook, Thunderbird and Evolution.

As you would expect there are no provisions in Microsoft programs to import from non Microshaft alternatives. Mozilla Thunderbird however is much better and does a first class job of importing mail, contacts and accounts from Outlook and Outlook Express. Likewise Mozilla Firefox imports favourites, history and cookies from Internet Explorer. The Mozilla programs have almost identical 'stores' for their information called profiles largely independent of the operating system in use so a transfer from Windows to Linux and vice versa is quite easy. In fact, under some conditions, it is possible to have a single profile accessible from both systems so you can move backwards and forwards. I understand that the latest versions of Evolution will also be capable of importing Microsoft Outlook .pst files which contain all the email, contact, calendar and task information.

Before getting to details of choosing our email program and the specifics of transferring information etc it is worth looking at how the main email systems work and how that can also help us in our transition, maintaining an audit trail and keeping various machines synchronised.

Email System Considerations - IMAP versus POP mailboxes (Background reading)

POP Mail: Most people understand how POP (Post Office Protocol) mail works, the incoming mail is delivered and held on a server at your Internet Service Provider (ISP) and, in the simplest case, you download it to your machine on demand (usually called a Send/Receive activity in your email package) and it is deleted from the server. This was fine when the typical user had only one machine in one place. Nowadays many people access email from home, their office, from a mobile and perhaps a PDA and Phone. This leaves ones incoming mail fragmented in many places. Most email packages and POP mailboxes therefore allow you to collect your email whilst leaving a copy on the server and many also allow one to just download the headers or a restricted amount of data in each email. This is a much better way of working as the downloads can be done on many machines and the eventual deletion from the server is done on one machine and only after it has been satisfactorily archived. It does not help with outgoing mail which has to be copied to another account or some other method to allow an audit trail.

IMAP: There is an alternative to POP mail called IMAP which stands for Internet Message Access Protocol. In this case the email is stored on the server and never downloaded automatically. One can create addition folders on the server so one can have a complete filing system on the server and available whilst one is online, on corporate systems there may even be shared folders accessible by many users. Email stored on an IMAP server can be accessed and manipulated from a desktop computer at home, a workstation at the office, and a notebook computer while traveling as well as ones phone/PDA, without the need to transfer messages or files back and forth between these computers. In the simplest case data is only transferred as required - when you select your inbox or a remote folder the headers are transfer - when you select an email the message body is transferred and only when you open an attachment is the attachment transferred You can of course copy anything to a local folder on your machine to work on it and when you finally send a reply (whilst online) it is saved on the server and accessible from any other machine. IMAP is at its best when you are on a Broadband always on internet connection - a GPRS/3G connection charged on data transfers is acceptable.

Webmail: This is really not a separate email system but a way of accessing existing POP or IMAP mailboxes directly via a web interface. The email stays in place unless you delete it or download it from a POP mailbox. It is ideal for access from an internet cafe, it is rather like using IMAP but without easy movement to a local folder system for archiving if used alone. Some email accounts are purely webmail however like Yahoo and Gmail

Implementation by ISPs: Many Internet Service Providers (ISPs) offer the choice of a mixture of POP or IMAP mailboxes - in some cases they are common boxes and the access protocols and port number used differentiates their use. Most POP and IMAP mailboxes are also accessible via a Webmail interface when you are in an internet cafe or on a friends machine. Even if you do not want to change completely without extensive trials it is worth setting up a single IMAP mailbox so when you are traveling so you can transfer mail from your 'mobile' machines via an IMAP mailbox to you home machines filing system via the IMAP mailbox without worries about the incompatible local mailbox and folder formats. You can also tidy up whilst traveling whenever you get a fast WiFi data link in an internet cafe. I do not feel comfortable solely depending on a remote server to store my emails long term but for a few months between archives it seems a very sensible way to proceed. With a GPRS/3G connection where one pays for data transfers rather than time online, it seems a very economical way to operate and most email packages allow one to download selected local copies for offline working as well as remote copies.

Some cautions: Both POP and IMAP protocols are define by RFCs but the implementation by email packages may not be rigorous when it comes to some of the more advanced features. It is possible that POP implementations of features used for leaving emails on the server for a fixed time differ between email packages and may be implemented locally or on the server so if you have different settings or ways those features are implemented between packages you may have a problem. A good way to start investigation is to see if the same mailbox can be accessed by POP, IMAP and Webmail without problems.

A Comparison of Outlook 2003, Evolution and Thunderbird

Firstly note that I have not even mentioned Outlook Express (now called Microsoft Mail) as the only thing in its favour is that it is free, familiar and built into all versions of Windows. For use at home with unlimited bandwidth and money to throw at virus and malware checkers and hiding behind hardware and software firewalls Outlook 2003 still has the high ground in my estimation. Outlook 2003 also allows synchronisation with my O2 XDA Exec and Pauline's Palm providing the backup of contacts, appointments, tasks etc.

The comparison for mobile use is less clear both under Windows where the choice is Outlook 2003 versus Thunderbird and on Linux where it is Evolution versus Thunderbird (or possibly Outlook 2003 running under Crossover Office). In the first case it all depends how much you use the functionality over and above email and address books and if you need to sync/backup a Windows Mobile Pocket PC. Thunderbird is faster and more secure in what it does but it does less - on email both offer downloading of headers or limiting size and both have good spam checks although the Outlook 2003 checker does need regular internet updates. Outlook has always had a superb archiving system and I have an audit trail back to 1996 and it is also easy to back up the single .pst file with everything in it. I still keep one dual booted system at home with Windows XP so I can use Outlook 2003 to provide a backup for the XDA contacts.

I put a particular emphasis on mobile operation when comparing the options under Linux as we spend a lot of time away from home and without access to Broadband with cost and speed implications. Evolution which is installed in Ubuntu as the default email package offers the most basic functionality in calendars, appointments, but seems very outdated - it is comparable to the earlier versions of Outlook patched to offer some modern functionality. In contrast Thunderbird has very much the same feel as Firefox, allows one much more control over how one downloads mail and various extensions (lightning for Calendars and Tasks and MoreFunctionsForAddressBook to strengthen contact management) add most, if not all, of the functionality found in Evolution and Outlook. The latest version 3.x has a tabbed presentation of email etc.

Whilst on the move I either download headers only or limit the download to the first few Kbytes - this is easy in Thunderbird and can be set independently for each account by Edit -> Account Settings -> Server Settings and Disk Space for each account. Downloading seems extremely fast compared to Outlook or Evolution. You have no such choice currently over headers or the ability to set download limits in Evolution.

Spam checking is very poor in Evolution and slows the downloading to a crawl, several seconds per email because the spam checking needs to do internet checking to be in the least effective. In contrast Thunderbird has been quickly trained and after a few hundred mails seems to work very well. I have been deliberately accessing a few old and spam ridden accounts for training and the last download identified 48 spams, missed 2 and had one real email identified as spam. That was better than Outlook 2003 did with monthly updates of its data base!

For periods of a few months away I leave all copies of all the mail on the server and then collect into the master system. Thunderbird also gives the ability to blind carbon copy all outgoing emails to a separate email box which ensures I can also pick up all outgoing emails on the home system.

I have therefore chosen Thunderbird, with two extensions installed, over Evolution as my standard email package.

Thunderbird

Install and Configure Thunderbird under Windows and import accounts, contacts and email

It may seem strange in a article on Ubuntu to start with a discussion of installing a program under Windows but this is the best way to transfer emails etc from Outlook Express and Outlook in Windows to Thunderbird in Ubuntu is via this step. The procedure is:

Configuring Thunderbird to leave mail on the server and for handling junk

Now we need to set up various options to match our way planned way of operation and for filtering junk email - this is appropriate to Thunderbird under Windows or Linux and you might as well do it all under Windows as these settings will be transferred to Linux as they are all stored in the Thunderbird 'profile' mentioned above.

Now we need to finish setting up the default junk email settings by

That all the options set up although you will see there are many more which can be useful to the way you chose to work especially if you are mobile and want to restrict the amount of downloads.

Thunderbird Profiles and Transfers from Windows Thunderbird to Ubuntu Thunderbird

The latest versions of Thunderbird contain all the emails and address book information in a single profile folder which is compatible between Windows and Linux of the same version number. On Windoz it is likely to be in C/Documents and Settings/Username/Application Data/Thunderbird/profile/sillyname.default/Mail/Local Folders/ On the Linux machine the profile is, by default, /~.mozilla-thunderbird/sillyname.default . Remember files starting in . are hidden in the file manager so use View -> Show hidden files. You can actually have several profiles in Thunderbird although there is no simple GUI way to access the Profile Manager built in. You can however open the Profile Manager from a terminal and create a new profile or use a profile you have copied from another Windows or Linux machine by

thunderbird -P

The profile manager is simple and self explanatory so the creation of a new profile using your profile from another machine is very simple and powerful and is also useful for restoring a backup. You can chose the profile to use in the future or chose to automatically open the profile manager every time you start Thunderbird from then on.

Thunderbird Version 3 has a number of extra entries in the profile folder which are automatically generated when an earlier profile type is opened. There are reports that it is still backward compatible but I have not tried and would advise caution in mixing like that.

Thunderbird Extensions are also stored in the profile folder and many extensions are not compatible between operating systems and between TB2, TB3 and TB5 so uninstalling all extensions before moving a profile is prudent although checks and warnings are usually provided and i rarely bother these days - I do tend to backup the profile fairly often.

Thunderbird Extensions

The general procedure for installing an extension in Thunderbird is:

  1. Download the Extension from the Thunderbird Add-ons homepage
  2. Start Thunderbird then -> Tools -> Add Ons
  3. Under Extensions Click “Install” button and look for the Extension Module (.xpi file) you just downloaded. Under Thunderbird 5 this is just a small button with drop downs.
  4. Wait for the count down, and click “Install”
  5. Now restart Thunderbird

The data and configuration information is usually stored in the profile folder.

The MoreFunctionsForAddressBook Extension for Thunderbird

I found a really useful extension for Thunderbird which enables one to see and use several of the hidden fields which are not yet implemented and I have installed it on all my machines. It is called MoreFunctionsForAddressBook and is now on version 0.6.8.1 which covers <TB18. It seems almost too good to be true. It adds in many of the functions which are missing in the Thunderbird address book as well as allowing display and use of some hidden fields such as Categories and Birthdays. You can also select and copy addresses and notes from the 'preview' and do searches which include extra fields such as categories and notes. It also extends the actions to handle exporting, sending and importing Vcards. I have done an extensive search looking for the catch but have not found any problems on the internet so far, if you search for problems you just get examples of the problems it solves! I still think that Outlook 2003 is the best overall email, contact management and calendaring package against which the opposition must be judged but Thunderbird with the Lightning and MoreFunctionsForAddressBook Extensions is now coming very very close even before you take into account the costs, security and other Microsoft issues.

  1. Download the Extension from the bottom of the MoreFunctionsForAddressBook page
  2. Start Thunderbird then -> Tools -> Add Ons
  3. Under Extensions Click “Install” button and look for the Extension Module (.xpi file) you just downloaded.
  4. Wait for the count down, and click “Install”
  5. Now restart Thunderbird

Other Extensions from the writer of MoreFunctionsForAddressBook

I was so impressed with MFFAB that I have tried and install as standard a number of other extensions.

Other Important Extensions in use

Sync Kolab: See below for more information under Synchronisation.

Scheduling - Calendars and Tasks

The lightning Extension to Thunderbird and Sunbird

Mozilla Sunbird is a cross-platform calendar application. It provides a full-featured and easy to use calendar application that can use a variety of standard internet calendar servers (eg WebDav and CalDav) and file formats such as iCal in parallel with its internal database.

Lightning integrates the Sunbird calendar with Thunderbird. Lightning is implemented as an extension making it much more tightly integrated with Thunderbird than the standalone Sunbird thus allowing you to easily perform email-related calendaring tasks. Lightning is by far the most important Extension to Thunderbird and changes it from a good email package to a full function schedule management application on par with Microsoft Outlook and Evolution in functionality. When this page was written it looked likely that future development would concentrate on Lightning as most of the use of Calendars and Tasks requires the use of email.

Installing the Lightning Extension

There are two ways to install Lightning in Ubuntu. The first is using Applications -> Add/Remove and search for Lightning which adds the latest version in the Ubuntu Repository system. The best way is to use the mechanisms built into Thunderbird. I believe the built in mechanism is best as this ensures you can chose the [latest] version especially if you intend to implement and of the sharing and synchronisation activities which will be covered later. It is possible to set up Thunderbird to tell you when updates are available or to automatically install them. It is best if it Lightening is the first extension installed.

The steps to use the built in mechanism to add the Lightning Calendar and Task Extension to Thunderbird are:

  1. Download Lightning Calendar module from Thunderbird Add-ons homepage
  2. Start Thunderbird then -> Tools -> Add Ons
  3. Under Extensions Click “Install” button and look for the Lightning Calendar Module (.xpi file) you just downloaded. The button is smaller under version 5.
  4. Wait for the count down, and click “Install”
  5. Now restart Thunderbird

The calendar and task data is normally stored in the extension although external iCal files (.ics) can be used

 

Sharing and Synchronising Email, Address Books, Calendars and Tasks

More about Thunderbird and other Mozilla Program 'profiles' and sharing data on a dual booted machine.

All the configuration and data for Mozilla programs are held in what is called a 'profile' and the 'profile' folders for all the main Mozilla programs look very similar. The profiles have what is called a salted name – that means it is randomly generated and 8 characters long so it is difficult for a hacker to identify. However the first default profile is in standard place and has an extension of .default and new profiles are reached by pointers in an .ini file one level up in a folder often called profiles. The easy way to find your profiles is to do a search for files with a *.default search. Ubuntu Hardy has a file search under Places.

I have set up all my Dual Boot machines to have at least one common Drive/Partition which can accessed from Windows or Ubuntu. I have made this a FAT32 partition because there were some concerns over the reliability of writing from Ubuntu to the proprietary format NTFS which Microsoft has developed. This has been overcome I understand and Hardy Heron and higher should be OK with an NTFS common drive.This FAT32 common partition has to be created before starting to install Ubuntu or created at the same time in an unused part of the drive using the Linux partitioning tools – do not try to modify a Windows Partition especially one used by Vista from outside of Windows. Vista allows one to shrink its partition to make unallocated space. Under XP I use an old copy of Partition Magic 8 to create and resize partitions.

This all opens up some very interesting opportunities as I recently realised. If one is running Thunderbird and Firefox under both Ubuntu and Windows then it should be possible to put their internal and configuration data (which is all in the 'Profile' folder) on the common drive and access emails, address books, calendars, browser favourites, cookies and history etc from both machines. This seemed a big step forwards and I used the technique of a common profile on a permanently mounted drive formatted as FAT32 so it was accessible both from Windows and Ubuntu for a period when I was still using Windows on a regular basis.

NOW A BIG WARNING
NEVER HIBERNATE IF YOU SHARE DATA

Hibernation works by writing the contents of memory to the disk and then retrieving it again when you turn back on. If you have been working on the same data or even access the same directories then the 'hibernated' data can and usually will overwrite it. It is possible to hibernate provided you go back to the same system but I have forgotten too many times so I turn off hibernation on both systems.

In Windows: Start -> Control Panel -> Power Saving.
I have covered it for for Ubuntu under Disabling Suspend and Hibernate in Ubuntu .

The only problem, other than hibernation, I have found in sharing the profiles for Firefox is with Extensions. In Windows AVG forces in a couple of extensions which are not available under Linux and there is an extension for Ubuntu when running in Linux which is clearly not available in Windows. I find that when I swap I get a popup saying extensions are being installed when I swap from one system to the other - this is to be expected but it would be nice if the messages could be suppressed. In general Extensions are likely to be a problem unless they are available and compatible in both Windows and Ubuntu. The Ubuntu Firefox Extension is added by Ubuntu to allow certain extensions and updates to be handled by the Ubuntu Repositories but in many cases it seems to inhibit correct operation of extension you add yourself in Firefox and many people inhibit it as a matter of course within Firefox.

Thunderbirds version 3 and 5 has a number of additional files in the profile folder and should only be dual booted with the same version.

The Sync Kolab Extension for Synchronising Thunderbird Address Books, Calendars and Tasks via an IMAP Server

Kolab is a Open Source Groupware Solution for Emails, Appointments, Contacts and more. It supports mixed client environments because of an open storage format. Kolab was created for the needs of the German Federal Office for Information Security (BSI) by contract in 2002-2004 and is used there in a heterogeneous environment. Kolab won the Linux New Media Award: Category 'Best Groupware Server' in November 2005. In other words it is a professional set-up. We are interested in only a very small part, namely SyncKolab which is the client which runs as an extension in Thunderbird. This can also be run using any standard IMAP mailbox to synchronise Thunderbird Addresses, Calendar Items and Tasks. Each item is converted into a standard Vcard or iCal format and stored on the IMAP server as though they were an email. When you synchronise only the headers need to be downloaded for checking for changes whilst the local copies are checked fully. Any changes are then 'mailed' to the server or the local copy updated. There is no limit to the number of clients that can be synchronised.

SyncKolab is extremely useful but has given me considerable grief and only recently has it been working to my satisfaction in its synchronisation of Tasks and even Address books have caused problems on occasions.

Data usage during a synchronisation: I was worried that there would be a significant cost in data but it seems to be small enough to be OK for mobile use. My address book has 850 entries and there are a small number of calendar entries - the total data flow the first time after connection when the complete set of headers has to be downloaded is about 50 kbytes and subsequent synchronisations reduce this to 10 kbytes, to this has to be added any cards which need to be updated but unless you have a lot of notes that will be under 1 kbytes for each. You might not want to do it every day whilst abroad but it is fine on a mobile in the UK.

Compatibility of Lightning versions: You need to have a compatible version of the Lightning extension installed which may be a problem if you have loaded a modified version for Ubuntu using the Package Manager - I have had to uninstall the package manager versions and use Thunderbirds internal mechanisms for extensions. The SyncKolab extension is very small and very easy to set up once you have made some folders on the IMAP mailbox. The configuration tool (Tools -> SyncKolab Options) takes you through a very obvious wizard and ends up in is a single screen with every possible parameter identified on your system and available to select via drop down boxes (IMAP mailbox, folders on the selected mailbox, address book and calendar). It can take a few minutes with an address book of 900 items like I have to download and check all the headers but that is a small price to have everything synchronised. I have customised the toolbar to have a Synchronisation Icon.

Odd characters: I initially found that I had 16 entries in an address book of 870 which always showed up as changed. I eventually tracked it down to certain characters and strings in the address book which had originally been converted from Outlook 2003 via a PDA! Tabs were the first problem and did not seem to be consistently handled in the conversion to and from vCards. The other problem was that some of the address stings had been concatenated by the previous conversions onto a single entry and some combinations of 'terminators' in particular | and ; gave problems. Once I found this out it was easy to check the notes for tabs and the address field for a concatenated entry on the problem entries and fix them. I am not sure if this is fundamental or a flaw in the particular implementation of vCard import/export - I suspect the latter. I believe all the problem had come from using cut and paste to fill in address cards in Outlook 2003 so now they are fixed I seen no future problems although I will be careful pasting into the notes field. I had similar problems with the address books when I started to use gContactSync -- see below

Complete Uninstall: I have found that when one has any serious anomolies it is worth trying the uninstall and reinstall proceedure on the SyncKolab Web Site. This seems more important after you first switch to Thunderbird 3 or use one of the 'nightly builds - the procedure is:


To completely remove all traces of synckolab:

if you have any problems at all the first thing to do is to uninstall and reinstall SyncKolab

Rebuilding IMAP Mailboxes: Another cause of problems is within Thunderbird as it is possible for the indexing files for mailboxes to become slightly out of kilter and Sync Kolab is very sensive to this. If you find that items do not get deleted then it is worth rebuilding the files in thunderbird. In particular I have found that problems with tasks failing to be deleted correctly in the past have been cured by rebuilding the indexes for the IMAP mailbox used for the Tasks - Right Click on the folder -> Properties -> General Information tab and click on Repair Folder.

Compatibility of Versions: It is normally better for all the machines to be running the same version of Thunderbird, Lightning and nightly version of SyncKolab

Options changes cause Downloads: I have found that any opening of SyncKolab Options sometimes forces a download of everything from the server so make sure you always Synchronise before changing any options to avoid recent local changes being overwriten - if you are just inspecting click cancel at the end.

Updates following change to Synckolab Version 2

The new version gets round a number of old problems with the status line displays and has a better display window and improved checking and handling conflicts but I found that I was suffering a number of strange hang ups and errors displayed in the error console. Many of these problems seem to be the result of the improved error checking which is showing up anomolies in some of my older data.

Firstly I should note this is following an update from Synckolab 1.5xx to 2.0 and many of my Contacts in the Personal Address Book (1011), Calendar items (121) and Tasks (13) go back many years and some seem to contain errors which were not found by version 1.5xx. It has been a long chase involving several machines and a number of profiles (including a new empty one) but the important steps in sorting it out were:

1. I made sure that the version on the server was up to date and backed up my profiles on all machines I was using.

2. I forced a complete local update (by changing the Options which forces a local update) - this got round the initial hang up and errors in the console and allowed the Synchronisation of my contacts to proceed. Two contacts had repeated conflicts whatever I did and finally had to be replaced.

3. I created a new local calendar which again forced a complete download when I changed the Synckolab Options to use it. A number of calendar items (6) had to be deleted on the server because they locked the system - three were complex repeated items and a couple contained strange code on inspecting the attaached .xml file. One I never found a cause but just got deleted. They were all old items so their loss was of no consequence.

4. None of the Tasks without a start date would synchronise until I created a start date. NOTE: Any finish dates I set were deleted after a couple of synchronisations which may be a bug.

It is now a considerable improvement on earlier versions - I must give many thanks to you for all the hard work put in by Nico Berger.

'Round Trip' test on Events and Tasks in the last issues of version 2 (nightlies) which are very favourable.

By 'Round Trip' I mean that I create or edit, say, a simple event on machine 1, Synchronise twice to make sure that there are no problems of alternating changes. Synchronise on machine 2, again twice for luck, make a minor change in the comment field to force the change to be propagated back to the first machine and check that everything is what one expects.

I have done that test progressively adding and changing each major field in turn for both Events (Calendar Items) and Tasks with no significant problems observed.

Caveats -

I continued my testing to check the Contacts/Addresses and all 'standard fields' (those visible in the Thunderbird contacts screens) are handled correctly and transfered between machines apart from:

Caveats

The tests above only confirmed that the fields are correctly transfered between machines. I have not checked that a change every one of the more obscure fields is detected. I advise any user who is dependent on a field to check themselves or make a habit of making a small change in the Notes: field - I try to add the date of each update to that field which also makes certain the change is identified.

So far I have explictly checked that changes are detected in the following less commonly used fields

The only field I have found where a field is transfer but a change is not detected is Categories, a useful field which is imported from Outlook into Thunderbird but is only visible in extensions such as MoreFunctionsForAddressBook

Test covering Support in Synckolab for the MoreFunctionsForAddressBook extension

I use the MoreFunctionsForAddressBook extension for Thunderbird which allows one to use some additional fields which are either specified for future use but currently not visible in the Thunderbird Address Book or are non-standard. The most important is Catagories which is supported within Thunderbird and imported from Outlook but can only be seen by using an extension. For the information of those of you who use MoreFunctionsForAddressBook, the following fields are supported:

The other additional MoreFunctionsForAddressBook fields are not supported and will be lost during synchronisation. These are:

SyncKolab Stable Version 3.0.0 and nightlies from 3.0.1

This is now coming into use in November 2012 but changes do not seem very significant over nightlies I have been using. Calendars however seem to have problems on the nightly I am using (3.0.1 of 21-11-2012) with TB17.02 and I have had to disable calendar synchronisation at present to enable the important address book synchronisation to continue.

gContactSync for use in Synchronising Thunderbird Contacts to Gmail and hence between many Machines and to a Blackberry.

gContactSync is an add-on (extension) that synchronizes contacts between Google (Gmail) and Thunderbird, and also supports importing contacts from Facebook, Twitter, and more. You can synchronize multiple accounts with multiple address books, and it fully supports regular and hosted Google Accounts. By default, it synchronizes the first four e-mail addresses, the first screen name and number of each type in Google and every attribute in Thunderbird (not all of these are visible in Gmail, however). It adds several new fields to Thunderbird for e-mail addresses, screen names, and numbers. Anything that is not synchronized should ideally be preserved but there are a couple of important exceptions where data can be lost in Thunderbird following a reset which results in a reload from the server. There are currently (December 2012) two versions 0.3.5 which is more stable but lacks the facilities to do what I require and the beta version of 0.4 (0.4.0b2 in Dec 2012) that I am using.

gContactSync can be set up to synchronise multiple Gmail Groups and in that mode expects the default groups of My Contacts, Friends, Coworkers and Families as was the case with Gmail Contacts until very recently when the defaults were changed to the single My Contacts group. See http://support.google.com/mail/bin/answer.py?hl=en&answer=97952 for further details of the changes and situation with existing contacts in those groups. Thunderbird does not implement Groups - there is a hidden field called Categories but that is not used by gContactSync and instead there has been a slightly uneasy shotgun marriage between Google Groups and Thunderbird Mailing Lists. I say uneasy because there are considerable differences between their Raison d'être - Thunderbird's Lists are a rather basic mechanism to add lists of contacts to emails and therefore expect an email address to be present whilst Google's Groups are much more flexible. Furthermore there are some long standing and intermittent open bugs in the Thunderbird Address Lists, in particular with respect to drag and drop. See https://bugzilla.mozilla.org/show_bug.cgi?id=282841#c8 as an example of problems going back over 6 years.

Preparation - Backing up and test set-up

When I started to use gContactSync I took a lot of precautions in backing up as when such synchronisation programs go wrong they can destroy lots of contacts or make multiple duplicates - I am not sure which is worse so better to play safe!

Installing gContactSync

I installed version 0.3.5 in the initial stages and almost imediately switched to the latest development (beta) version - go to gContactSync , scroll to the bottom of the page and expand Developer Channel where you will find the link to install the latest development version. Once you have downloaded a development version it will automatically update to the latest development version.

Setting-up gContactSync.

It is important to do the initial Synchronisation correctly. Either the address book on the PC or in gmail preferably needs to be empty to avoid any duplicates and I found that by far the best way and the easiest way to just create a new empty address book in Thunderbird (Open the Address Book -> File -> New -> Address Book) or even easier create it within gContactSync and then synchronise to a Gmail account with an existing Contacts list. You can then copy your existing contacts across to the new Address Book in sections and synchronise to get them into gmail if they are not already there. There are various choices on the setup screen for an Account in the Groups and Contacts to Synchronise dropdown and one needs an appropriate choice.

During all my early tests and for many months I thought that the All Groups and Contacts was the best option but I am currently experimenting with more success using just the My Contacts option. This has the advantage of automatically adding every item in the Thunderbird address book to the Google Contacts 'My Contacts' Group without having to create/use a flaky Thunderbird Mailing list. Your choice will have to depend on whether you already use Google Groups, use Thunderbird Mailing Lists and/or are synchronising a PDA/Smart Phone. I had so many troubles with Thunderbird and Lists in the past that I have never added them to my Personal Address Book (PAB) and have just used them when absolutely essential in copies of my PAB so my choice is more obvious as I do not need to ever synchronise them. I however intend to look again into the behaviour of Thunderbird Lists in more detail and will initially write up what I find in a Diary page.

gContactSync Testing with Beta Version 0.4.0b2 and higher (Dec 2012)

One is prudent to run some initial tests when you start to use an new synchronisation program and not to risk your master address list (Personal Address Book) or arguably ever - run gContactSync on Personal Address Book (PAB). There are a few extra steps I went through to ensure that all the fields are correctly synchronised as gContactSync is forced into a making a few actions which seem a little idiosyncratic at first sight but are actually sensible and others are work-rounds for shortfalls and bugs in the Google APIs (Application Programming Interfaces). It is worth reading http://www.pirules.org/blog/ which is far more than a blog as it has the various bits of information that help one work out why some things have been done the way they have.

You will notice that once gContactSync is install the contact editor has a lot of more tabs as gContactSync (and MoreFunctionsForAddressBook if you use it) both add their own extra tabs. You will also note that address book items without an email have often have a dummy address added - without this a contact without a email address will not synchronise - the program description implies they have this dummy email address added by gContactSync to get round a Thunderbird problem with address lists items without email addresses causing a freeze/crash. One outstanding anomaly if you are set up to synchronise all groups and Contacts is that one synchronises with a new contact (or one that has been copied into the address book) it does not end up in 'My Contacts' in gmail contacts but in 'Other Contacts' which does not transfer to the Blackberry and likewise deleted contacts are also transferred to 'other contacts' from Thunderbird - I find that for my appliation editing is best done on my Blackberry. This seems to be a known feature rather than a bug but it would be nice to have an option available.

You may wish to follow the same route as I did to test gContactSync and the various workrounds I will come to latter. What I did was to set up an extra two address books so I could test using synchronisation between these two contact lists rather than use separate machines. I used a 'spare' gmail account I set up earlier for this purpose. This technique worked very well. I started off by creating the two address books then added the two new accounts to them. I turned off all the fancy things with phone numbers in the preferences windows so I kept as close to Thunderbird as possible.

You need to have Address Synchronisation turned on (this is off by default on version 3.5 but should on in 0.4.0) as there were and still are various differences in the way Thunderbird and Google store addresses. Thunderbird stores what are know as structured addresses, which are the individual components of an address (street, city, state, zip code, country, etc.) while Gmail originally only stored a formatted address which contained the entire address in a single field. Google now automatically converts addresses between these two formats hence version 0.4 has it on by default.

The basic test was to create a new Contact and make sure it is transferred between address books and then fill in all the standard and extra fields and see that they were all transferred correctly - that is not always the whole story as the changes also have to be detected so the test really needs to be done field by field. Many of the fields are common between Google Contacts and Thunderbird addresses but some are not consistent and more likely to give problems. These include Phone numbers and fields that make up an Address. There are also a set of extended fields in Thunderbird some of which can only been seen and changed by Extensions such as MoreFunctionsForAddressBook and some are added by gContactSync and these also needed to be investigated.

I should also describe my set up as there may be interactions to sort out. I am (Dec 2012) using Thunderbird 17.02 with the Lightening Extensions to give Calendars and Tasks - an almost universal set up. In addition I have several Extensions which are to do with address books:

Unexpected results in the tests

  1. New Contacts have to be synced before they are added locally to My Contacts then synced again if you are using options synchronising groups to lists. Not sure if this is a bug or an anomaly
  2. Address Line 2 is a real problem - it is not synchronised as it does not exist in Gmail Contacts but it can still shows a conflict if it is changed and updating of everything on that card can be blocked. A quick and dirty workaround is to force synchronisation in the case of a conflict to be from the server to local which at least allows other changes to be propagated even if the second address field does not sync or see below for better solution. This is a bug in my book.
  3. Contacts without a email address will not synchronise - the program description implies they should have a dummy email address added by gContactSync to get round a Thunderbird problem with address lists items without email addresses causing a freeze/crash but this does not occur if the field contains and invalid address or a text string such as none or is cleared locally. Clearing the field online in Google Contacts leads correctly to the dummy address being created. Not sure if this is a bug or an anomaly

Summary of supported fields and synchronisation results using gContactSync (and SyncKolab) TB to TB via Google and Blackberry to Thunderbird via Google using Google or BIS Sync on the Blackberry.

The initial table has been extended to include a comparison of gContactSync (SyncG') with SyncKolab (SyncK' ) which I also use and similar results from BIS Sync for Blackberry to investigate the wider integration of address books.

y = supported or sync works . N or n = Not Supported or sync fails. h = hidden field. C = Column can be displayed. S = Cards can be searched for this field.

Some shortfalls and need and consequences of Workrounds in the various programs are highlighted.

Field
Basic TB fields
MFFAB fields gC'Sync fields SyncG' SyncK'

BB fields

BIS Sync
or GoogleC
First Name
y
y
y
y
y
Last Name
y
y
y
y
y
Display Name
y
C
S
C
y
y
Prefer Display (Tick)
y
y
N
Nickname
y
C
y
y
y
y
Email
y
C
S
C
y
y
y
y
Additional Email
y
C
C
y
y
y
y
Screenname
y
C
y
y
Prefered Format
y
y
y
Allow Remote
y
y
Y with 3.0
Work
y
C
C
y
y
y
y
Work 2 (BB tel field)
             
n
 
y
y
Home
y
C
C
y
y
y
y
Home 2 (BB tel field)
             
n
 
y
y
Fax
y
C
C
y
y
y
y
Pager
y
C
C
y
y
y
y
Mobile
y
C
C
y
y
y
y
Mobile 2 (BB tel field)
               
y
y
Home Address 1
y
C
y
y
y
y
Home Address 2
y
N/Y
y
y
Merged into Home Address 1 on transfer to Google
Home City
y
C
S
y
y
y
y
Home State
y
C
y
y
y
y
Home Zip
y
C
y
y
y
y
Home Country
y
C
S
y
y
y
y
Home Web Page
y
C
y
y
y
n
Birthdate
y
Birth Year
y
y
y
n
Title
y
C
C
y
y
y
y

Department

y
C
C
y
y
y
y
Organisation
y
C
C
S
y
y
y
y
Work Address 1
y
C
y
y
y
y
Work Address 2
y
N/Y
y
y
Merged into Work Address 1 on transfer to Google
Work City
y
C
S
y
y
y
y
Work State
y
C
y
y
y
y
Work Zip
y
C
y
y
y
y
Work Country
y
C
S
y
y
y
y
Work Web Page
y
C
y
y
y
Custom 1
y
C
S
y
y
User1
n
Custom 2
y
C
S

y

y
User2
n
Custom 3
y
C
S
Y/N
y
User3
n
Custom 4
y
C
S
Y/N
y
User4
n
Notes
y
S
y
y
y
y
Photograph
y
y
y
y
n
Categories
h
y
S
N/Y
y
y
n
Anniversary
y
Y
y
y
n
Additional Emails (5)
y
N
N
Spouse name
y
N
N
Card Mod date
y
?
N
Extra Fields (10)
y
N
N
Phone 6 and 7
y
Y
N
Additional Emails (2)
y
Y
N
Additional Screennames (5)
y
Y
N
People (4)
y
Y
N
Other Phone (BB tel field)
y
y

The above are the results of actual tests rather than what the manuals say should happen. Most have been 'Round Trip tests. By 'Round Trip' I mean that I create or edit, say, a contact on machine 1 (or address book 1), Synchronise twice to make sure that there are no problems of alternating changes. Synchronise on machine 2 (or address book 2), again twice for luck, make a minor change in the comment field to force the change to be propagated back to the first machine and check that everything is what one expects. The fields marked N/Y and Y/N are because I have used a fudge to allow synchronisation of HomeAddress2 and WorkAddress2 between machines running Thunderbird to preserve a large number of my addresses which have a second line (~170 in 1000) and similarly for Categories which is a hidden field but available through some Thunderbird extensions I use. Details are in the next section:

Solving the Second Line of Home and Work Addresses problem and Synchronising the Categories field.

The address field problems occur because the fields used by Thunderbird and by Google Contacts do not match. Google does now support 'fixed' fields like Thunderbird for all Addresses but one field (the second Thunderbird address line has no equivalent). Google Contacts instead has a first address field which supports multiple lines. This gives a problem as the second address field tends to get lost during transfers between machines running TB or during a reset which reloads everything from the Google Server.

gContactSync however has a facility to synchronise 10 extra fields which do not exist in Google Contacts but are fields that gContactSync should sync with other Thunderbird clients. These fields are not visible in Gmail; only gContactSync sees these fields and knows what to do with them. The names must be valid fields in Thunderbird. It's main purpose is for users who synchronize a single Gmail account across several computers (such as businesses who synchronize business contacts to their employees' PCs). There are currently are ten of these extension/extended fields which are preset to useful functions such as 'preferred email format' and the 4 Custom fields. In version 3 they were on a tab which has been removed in version 0.4. The feature is still there but it requires editing preferences in the Thunderbird Config Editor to change the fields that are synchronized.

I have decided that preserving my information in HomeAddress2 and WorkAddress2 is more important than synchronising Custom 3 and Custom 4 that I do not use at present so my fix is:

Edit -> Preferences -> Advanced -> General tab and click Configuration Editor and read warning then continue and Search for extensions.gcontactsync.extended and right click -> modify and change extended9 value to HomeAddress2 from Custom3 and extended10 value from Custom4 to Workaddress2

I have also used the same mechanism to enable me to synchronise the hidden but useful Categories field the same way by,

Edit -> Preferences -> Advanced -> General tab and click Configuration Editor, read warning and ignore, then continue and Search for extensions.gcontactsync.extended and right click -> modify and change the extended6 value to Categories - this replaces something I have never heard of called PopularityIndex.

This has to be done on every copy of Thunderbird you are using (in fact within every profile if you have multiple profiles) before you do any serious synchronisation to avoid loss of information from the second line of Addresses or the Categories.

This may sound complicated so I will show a screen dump of the end result which, after making changes to support Categories, HomeAddress2 and WorkAddress2, looks like this:

Changes to Extended Fields

This change allows the second address fields to be synchronised between Thunderbird address books but not in a way accessible to Google Contacts and at the additional expense of loss of synchronisation of Custom 3 and Custom 4 - I do not use them but that might be a problem to others. I have put a posting in the conference suggesting more such fields are made available.

Summary of Configuration changes made to gContactSync in Preferences.

When these configuration changes have been made the Preferences screens should look like this:

Main tab of Preferences MenuInterfaces tab of Preferences Menu

Setting up gContactSync Accounts

I have rather skipped over how to set up gContactSync Accounts which is where the form of likage between Gmail Contacts and Thunderbird Addressbooks is defined. I will now do so in some detail as I initially did not undersatnd exactly what the various settings implied and the help and howto information is sparse. Firstly one must remember that a Gmail Contact Group can be Synchronised to either a single addressbook or to a TB List within an Address book. A good default is to avoid all use of Thunderbird Lists which are flaky to say the least and to Synchronise only with the 'master' Group 'My Contacts' where all your normal contacts should reside. This is exactly the synchronisation set up used by most PDAs and Smart Phones when they synchronise with Gmail Contacts. This is certainly what I have found best and trouble free for me when synchronising my Blackberry to Thunderbird and Thunderbird to Thunderbird between up to 7 machines.

This is also the default already provided if you touch nothing under Groups and Contacts to Synchronise as the dropdown is already set to My Contacts and you do not need to click Find all Groups for this setting. Just use the New Address Book button to create a new address book with a name such as GCS shown in the example below and the click Add a New Account link to get to a window where you can enter your Gmail Account and Password details. At the end it will look like this for the GCS address book which is highlighted although not very clearly.

Screenshot

One does not have to synchronise to My Contacts - you might have a Google Group called Newsletters which you want to Synchronise to an address book. Now is the time to Click the Find all Groups button and you will see below that the extra Group called Newsletters is in the drop down list and has been selected instead. Click save changes and you are set up.

Screenshot

In the case of Synchronising with Newsgroups you might want to make sure that it is never updated by accident from some machines. Check the Show Advanced Settinngs box to give access to another dropdown menu where you can pick the direction for synchronisation.

Screenshot

The last configuration I will discuss is also the most flexible where All groups and contacts are synchronised. In this case each Google Contact Group is synchronised to a matching Thunderbrd List. I used it for a period but I did not need the flexibility and have never used Thunderbird Lists very much as I have never found them consistent regardless of their use with gContactSync. If you already use Thunderbird Lists successfully or already have a lot of Google Contact Groups you should try this but make backups. Lists Items have to have an email address and gContactSync will add a dumm one when required which is visible in Thunderbird but should not be transferred to Google Contacts. gContactSync will create Lists for My Contacts, Friends, Family, Coworkers and any Groups you have created in Google Contacts and vice versa automatically for your Thunderbird Lists.

Screenshot

Cleaning up old Thunderbird Address Books

My Thunderbird address books have a long and chequered history. They started life at least twelve years ago as address books in Microsoft Outlook and have suffered a number of syncing programs including Microsoft Active Sync and SyncKolab. I have two main address books, the root one formed the basis of that synchronised with the Blackberry and was transfered to the Blackberry via an LDIF export, import into Microsoft Outlook Express then into Outlook 2003 (2003 does not read LDIF or import from TB directly) then was transfered to the Blackberry using the Blackberry Desktop Software which Syncs Outlook 2003 and the Blackberry. It was then Synchronised and Backed up using Google Sync to Google Contacts and is now back on in TB via gContactSync. The amazing thing is that it mostly worked after all that. In summary:

  1. Some contacts without an email address have had a [No Email address found] string added to the first email field which prevent gContactSync from working as it can not add its random dummy email address to the field which is required to make TB lists work reliably.
  2. The original address book has a large number of invalid Birthdays and Anniversaries. The fields seem to need either a sensible date or be blank otherwise it causes errors in the Google APIs (Birthdays) or in gContactSync (Anniversaries) which show in the various error logs and prevent transfers to the server.
  3. Comment fields with characters that could not be read are replaced by err:501 or err:509 during transfers
  4. Cards with duplicate email addresses anywhere elsewhere in the address book are not synchronised or transferred.

The problems above should not affect most people but if you want to see some the somewhat brute force solutions I ended up using on a large address book Click here to to expand this topic

How to get a new card into the Gmail 'My Contacts' Group when synchronising Groups and Lists

It is known that Thunderbird has severe problems with Lists and, in particular, Drag and Drop into them which have been going on for over ten years judging from some bug reports. What is worse the problems seem to be dependent on any changes in version so one week you may think something is working and the next it has again reverted. There are some further quirks when one is synchronising using the 'All Groups and Contacts' setting in getting cards into the My Contacts Group which is essential if you are also synchronising a PDA/Smart Phone. Once they are in the Groups and Lists updates to cards seem to be fine.

What happens is that when one creates a new card or drags one across from another address book and synchronises the card only appears in Google in the other 'group' which I think is to be expected. If one also drags it into the My Contacts list in TB before the first synchronisation it still only ends up in 'other'. If however one does a synchronisation as soon as you have created a new card or draged one across from another address book ie before adding to any lists and then drags the contact into My Contacts list in TB it correctly transfers to My Contacts in Google on the next synchronisation. Trying to get round an initial mistake and not synchronising at the correct time by deleting the card from the local list, syncing then try to add it back does not work and it does not show in the list until after a restart of TB - there must be a shadow of the old entry in the list until Thunderbird is restarted.

This requirement on timing of the synchronisations and problems with deleted entries is not a logical behavior and seems to be a bug but it is not clear if it is with Thunderbird, gContactSync or even the Gmail Contacts API. I have seen somewhere that there is an automatic compaction of Thunderbird address books and lists (at shutdown or computer restart??) which does not always reset all the pointers correctly especially after an error has occured and that in the limit one sometimes has to delete all the contents (or create a new address book) and download again from the Gmail Contacts (ie use the reset function) or reload from a backup (But does a copy of the .mab file still hold the errors??).

I do not like this unpredictable behaviour. At present I am using accounts set up to just synchronise the Google My Contacts Group and I do not have any Contact Lists on the Address book it is synchronised to. I also have an account for All Groups and Contacts with a different address book on one of my machines to continue testing. It may not be a good long term solution to be duplicating every Synchronisation on a machine but it does make testing very easy and gives an element of backup against one-off failures.

Conclusions on gContactSync

gContactSync works surprisingly well once one has done the one off setup and has a clean address book - almost all my problems were not caused by gContactSync itself but by the data fields in my original address books - it is possible that better error checking and handling could be included. I have been using it full time for 8 months between several Linux machines and my Blackberry without any problems. I continued to use SyncKolab in Parallel during this period to synchronises my calendars and tasks and to provide an independent address book. If I did not have an IMAP mailbox available to support SyncKolab I would now be very happy to solely use gContactSync for Thunderbird Contacts and the 'Provider for Gmail Calendar' add-in for Calendars. Yes there are a few quirks and some need for configuration to preserve and access some fields and to make new entries easily but that is a small price for what has been a very robust piece of software. I do however find it worrying that the only pieces of software I know of to carry out this very important task are written and supported by single developers working in their own time whilst Google is withdrawing the support it provided from its competitors such as Blackberry. We all owe a great vote of thanks to such people as Josh Geenen.

Using the Provider for Google Calendars Add-on to Synchronise a Gmail Calendar (and hence a Blackberry) with Lightning

If you need both read and write access to a Google Calendar from Thunderbird/Lightning the best way is to install the Provider for Google Calendar from addons.mozilla.org. This gives a very simple way to get an extra calendar in Lightening/Thunderbird which is synchronised with the Google Calendar and hence your Blackberry if you are using BIS with a Gmail account with synchronisation enabled. There may be some anamolies if you have very fancy repeats or alarms but all normal calendar items and functions are fine.

Get the extension from https://addons.mozilla.org/en-US/thunderbird/addon/provider-for-google-calendar/ - Download the file by right-clicking on the link and choose Save-as. Then in Thunderbird with Lightning installed, choose Extras/Tools > Addons and click on the install button. Point it to the downloaded file and install the extension.

To access your calendar data, you need to retrieve your private XML URL from the Google Calendar UI.

1. Open your Google Calendar
2. Click "Settings - Calendars"
3. Click on the Calendar you would like to use in Lightning
4. Click on the private XML button and copy the link that is shown.

The private link will look something like: https://www.google.com/calendar/feeds/username%40gmail.com/private-1ba218e6a25bfc32b25a4eb3f9ee6d96/basic

After installing the 'Provider for Google Calendars' extension you can create a new calendar by File -> New -> Calendar and choose "On the network". Then pick "Google Calendar" and paste or enter the URL to your calendar and check the cache box. Click "next" and choose a name and color. Click "next" and a popup appears in which you have to enter your login (with or without the @gmail.com part) and password and choose whether you want Lightning/Sunbird to save your password. Choosing "next" creates the calendar and provides you with two-way access to the google-calendar. There will be a new toolbar item in Calendar view called Synchronise which synchronises your Google Calendars on demand as well as when you start and at 30 minute intervals.

Sending TXT (SMS) Messages from Telephones and USB Dongles

Introduction

I have been looking for a way to handle SMS messages under Ubuntu, in particular when using my USB Dongles which are designed for Mobile Broadband but use SMS messages to purchase additional time or packages, check usage and other housekeeping associated with their use. In general I have been using Vodafone in the UK and NZ for Broadband with a variety of SIMs and tariffs. Vodafone provide a comprehensive 'dashboard' under Windows which I have had to access on a dual booted machine occasionally or to do my 'accounting' via the web. I want to avoid having to use Windoz as we are moving to pure Ubuntu machines without dual boot. One solution is to use the very complex to implement Vodafone Dashboard for Linux I covered in the Page on Mobile Communications.

SMS Tools under Linux - knokii and friends

The main tools I initially found available within the Ubuntu distributions are based round the package gnokii. The useful interfaces are Xgnokii which is a GUI front end to gnokii and gnome-phone-manager which is an even simpler applet which sits in the tool tray and alerts you to incoming SMS messages and allows you to reply or write new ones. Many of the USB Mobile Broadband Dongles have several Serial USB Ports so that they can provide information on, for example, signal strength whilst connected via an alternative port.

knokii configuration

Although you can install knokii using the Synaptic Package Manager you still need to copy a configuration file /etc/knokiird into your home directory, rename to .knokiirc and configure for your connection (path to the port used for SMS, often /dev/ttyUSB1 ) and model to AT (AT command set interface) at a minimum. You can at re- configure the gnome-phone-manager within the package but you do still need knokii to be set up first and it is desirable to do some checks using gnokii before using anything more sophisticated.

In trying to get to grips with what was going on I found a very full explanation and tutorial which provides many of the basic AT codes and demonstrations of how you can use them to test the facilities of a GSM phone or dongle to send and receive SMS messages at the Developers Home SMS Tutorial. It is very comprehensive and runs to 66 pages, it gave me a much improved understanding and enabled me to receive and look at an incoming series of TXT messages quite easily.

Example - ZTE manufactured Vodafone K3565-Z USB mobile broadband stick.

I found using these that there seem to be two responsive serial USB ports for the ZTE modem which are /dev/ttyUSB1 and /dev/ttyUSB3. I have found the SMS facilities can be run in parallel with a Mobile Broadband connection by using port /dev/ttyUSB1 for the SMSs and /dev/ttyUSB3 for the broadband connection.

Tests using gnokii

gnokii --identify is the first test of any stick - the important part of the output from the K3565-Z was

Received message type 06
IMEI : 35651703397xxxx
Manufacturer : ZTE INCORPORATED
Model : K3565-Z
Product name : K3565-Z
Revision : BD_P673A2V1.0.0B09
Serial device: closing device

The ZTE manufactured K3565-Z uses /dev/ttyUSB3 for the internet connection. I have been able to send and receive txts using /dev/ttyUSB1 on the ZTE stick and also on the original Huawei Vodafone Z3565 which uses /dev/ttyUSB0 for Internet.

knokii can be used to read messages from the stick. The gnokii terminal command to read all the current SMS messages from the phone is:

gnokii --getsms SM 1 end -a Desktop/SMS_Messages.txt
This reads sms messages (gnokii -- getsms) from the Simcard (SM) from message 1 to the end and appends (-a) to a file Desktop/SMS_Messages.txt

You can delete a message by

gnokii --deletesms SM 4
Which deletes message a message gnokii -deletesms stored in SIM (SM) location 4 .

gnome-phone-manager Applet

The gnome-phone-manager applet is basic but easily installed by the Synaptic Package Manager. This allows one to send and receive SMS messages from a GUI interface accessed via an Applet which resides in the tool tray once it has been run from Applications. This is enough to enable me to interrogate the NZ phone balance by BAL to 777 and MYADDONS to 756 as well as various purchase of bundles and top-ups. The SMS messages seem to only be displayed when the applet is running (and you have a signal) but this is fine for quick interrogations of the balances and any other interactions where you expect a near instantaneous response. It means that one can do almost everything without a Vodafone Dashboard and Windows or use of the internet. A better SMS package is still needed for checking for 'old' SMS messages and cleaning up. I tried Xgnokii which works to some extent but the display is odd and it does not access everything. This led me to try the Wammu telephone manager.

Wammu Telephone Manager

I finally found a package which does what it is advertised to do as far as sending and receiving SMS messages and claims to work with all AT command compatible phones including USB Mobile Broadband Dongles. Wammu is a GUI front end to Gammu which had its early origins in knokii. It can be installed by the Synaptic Package Manager and there is also a Launchpad PPA available so you can get the latest versions and have them automatically updated - I used the straight version from Jaunty for these tests.

When I ran it with the Vodafone K3565-Z stick and NZ SIM it took me to various set up screens where I used the manual option setting it to /dev/ttyUSB1 and for at communication. You first need to use the menus to connect to the phone (Phone -> Connect) or set it to auto-connect and then you can

  1. Use Retrieve -> Messages to access all the SMS messages that are still on the phone - it retrieves both those in the phone or the SIM memories.
  2. You can send a single or multipart message by Create -> Message
  3. If you right click a message you have options including delete.
  4. You can Backup the messages in many ways including sending them to an IMAP server (only tested to files)
  5. You can Retrieve Calls and Contacts (tested for preset Contacts and Outgoing calls)
  6. You can set an option to automatically connect to a phone when you run the program (Wammu -> Settings -> Gammu tab and tick box) but not to Retrieve Info or Messages.

So far everything I have used has worked and it does everything I need to handle SMS messages on my Mobile Broadband USB sticks and it should be able to do much more via a Bluetooth to my phones as it also handles contacts and calendars. I have not tried it with Windows Mobile but do not expect that to work as Microshaft rarely implement any common standards as one would expect.

Wammu PPA and Lucid: I have started using the latest versions of Wammu, and Gammu which underpin it, from the Personal package Archive (PPA ) repository of the writer because I found that the versions in the standard repositories with Lucid throw an exception and crash when started (Gammu version 1.26.1-2). I tried a total uninstall and re-install without any effect.There was nothing shown on googling or bug reports butt it seems liikely it is a problem with the changes to the latest versions of libraries and backward compatibility. I therefore switched to the latest versions of Wammu/Gammu which take this into account by adding the Personal package Archive (PPA ) repository of the writer by running the following in a terminal:

sudo add-apt-repository ppa:nijel/ppa && sudo apt-get update

and then doing an update using System-> Administration -> Update Manager which finds the latest versions and updates everything.

It now works fine again and I assume that the latest versions (Gammu version 1.27.1-1) will eventually find their way into the Ubuntu repositories but until then I recommend using the PPA and have added it to my standard installation script.

Wammu in Natty: This is Wammu version .35 and gammu 1.28 so a PPA is not required. All the fixes for the Systray, Bluetooth Startup and rfcomm creation are required as one would expect if you are using Bluetooth. Otherwise there seem to be no changes.

Shortfalls in Wammu: Wammu does not have a mode that I have found to automatically monitor for the arrival of new SMS messages and you need to run the gnome-phone-manager to do that. It does not automatically download messages, contacts etc when you start it so you always need to retrieve them manually. The lack of monitoring for new messages is a nuisance as you need to run the gnome-phone-manager if you expect messages then change to Wammu for anything more sophisticated. The manual downloading is not a problem once you realise but it would be better to download the first time when you start up. Signal strength monitoring seems to be odd and works on some USB sticks and not others so the ability to set up the AT string to check in settings would be an advantage. Overall it is a very good tool with a few irritations which I hope are fixed in future versions.

Before You Leave

I would be very pleased if visitors could spare a little time to give me some feedback - it is the only way I know who has visited, if it is useful and how I should develop it's content and the techniques used. I would be delighted if you could send comments or just let me know you have visited by sending a quick Message to me.

Link to W3C HTML5 Validator Copyright © Peter & Pauline Curtis
Fonts revised: 28th April, 2021