|Home||Uniquely NZ||Travel||Howto||Pauline||Small Firms|
Diary of System and Website Development
Part 6 (January - December 1999)
Time for a review It is now three years since we specified the first PC we would own rather than use for work. We still have it as our main machine, the hardware a little extended, but still adequate. We still have the same 28 K baud Modem as our main interface to the outside world and the original philosophies behind our choices of system and software have held good. The major addition has been our Web Site which has now been running for nearly three years and now plays a central role. The number of hits has steadily increased and topped 2000 this morning which was one reason for thinking it was time for a review. The other was that we are away from home and working on the Libretto and Pauline made the comment that she was having increasing difficulty in keeping up and understanding our software and systems which has brought home the rate at which changes are still taking place and the differences in functionality over three years.
Operating System changes: Three years ago Windows 95 was new and certainly not in use at our work. It has been upgraded first to Service Release 1 (bug fixes) and then to OSR2 which made minor changes when it comes to connectivity but little visible to the user. The most important changes to the operating system have been the Active desktop - installed as part of the Internet Explorer 4 package. These changes mean our main system is almost the same as Windows 98 in functionality. This is a major difference between the Dell and the Libretto where we have not dared load Internet Explorer 4 or the active desktop - no nice row of icons on the toolbar for all the common activities and no nicely configured Start Menu.
Computer changes: The main machine has not been replaced but the memory has progressively been increased from 8 Mbytes to 48 Mbytes which has transformed the performance and mitigates the processor only being a 120 MHz Pentium. We now have a tiny Toshiba Libretto with 16 Mbytes and a 75 MHz processor which does seem slow in comparison but it is adequate for when we are on the boat and otherwise on the move.
The software review is still in draft and will have to when we return
I also installed the Revel1.1 utility which enables one to see what the passwords behind the rows of ***** are. it is very useful if one forgets passwords or has to do upgrades on somebody else machine but it a real security risk as the executable is only a few k and runs happily from a floppy. I have caused consternation when I have demonstrated the security risk of saving passwords to several people.
Freezone: The same cover disk had information about another free Internet Service Provider called Freezone. This one is very quick to sign up to, has 20 Mbytes of web space and will also cheaply register and maintain a Domain Name for one. I will probably add it to Freeserve and Telinco in my notes for OU students when I have checked it out. Freeserve still remains my prime TCP/IP connection as it is very fast and reliable.
Web Site: The web site has been mirrored on Freeserve for a long time but the main entry points have had to be transferred and warnings put on the CompuServe site as their servers have been so unreliable recently. Postings have been made into many of the OU conferences.
Egg Internet Service: Having accessed the Egg services it seemed logical to investigate their ISP service, an attraction being that it offers some additional connectivity via local dial up numbers whilst one is abroad. I could find no way round loading Egg Internet Service CD to register and was rash enough to run the install, fortunately only on our development drive as this caused major problems. It seems their Launcher is completely incompatible with Internet Explorer 5 and takes down the complete Browser system. It is all written as a Separate Guide to Egg's Internet Service - another free ISP service with Email and Web space.
Barclays Banking Service and Internet Explorer version compatibility issues. The next blow was that Barclays Online Banking was also not compatible with IE5, at least they knew and warned one and we now have IE4.01 on the Libretto! In the last few weeks they have made it compatible with IE5 only to find a bug in the Microsoft Java Engine so there were another set of warnings that one needed a 6 Mbyte download to update the Engine and a new set of foundation classes. The end result was what we wanted and we can see our Visa bills on the Barclaycard site, and we did a trial download from Egg and we set up a new payee and transferred the money directly to Barclaycard on the Barclays site. We have yet to see if the Mobile link will be fast enough to be practical.
SMS - Mobile Telephone - message integration: Most Digital mobile phones are able to receive Short Message Service (SMS) messages - they are rather like messages on a pager and can be up to 160 characters long and can be looked at on the telephone display. They are stored for up to 17 days at the mobile service provider for typically 17 days and are then sent to the phone. Phones will typically store 10 on their SIM cards so you have to delete them after reading to avoid a backup at your service provider. Normally they are used between phones but they can also be sent in various ways via your service provider's messaging centre and their are programmes which dial into your provider (at mobile rates) and "upload" messages. One I have had recommended but not used is SMSMaster which allows messages to all networks - details at http://www.haigh1.demon.co.uk You can also try http://users.breathemail.net/a.underwood/cellnet.htm for information on Cellnet.
BTcellnet Genie: BTcellnet have brought out a number of services which offer a number of ways of integrating Web, Email and SMS messaging. Unfortunately, although they overlap to some extent, they have to be registered for separately. A must if you have a Cellnet phone is Genie at http://www.genie.co.uk. This allows one to register an email address for short messages (under 160 characters) to be forwarded as SMS messages to your telephone. There are no costs involved (beyond that of sending an email). If you are not using the phone you can log in at the Genie site and get them forwarded as email to you instead or just held as copies on the Genie site. Copies are held on the site in any case. You can do far more with your phone, for example, you can set up a portfolio of shares and get the value sent at the end of trading every day and even set limits on particular shares and get an SMS message sent if they go outside set limits. You can also send SMS messages to other Cellnet phones using a web type email interface which has a simple address book. Again I can find no associated costs.
BTcellnet U.Genie ISP service: There are two separate parts of the U.Genie service which again need a separate registration. The first is the U.Genie Internet Service Provider - this offers all the usual facilities including 10 Mbytes of web space with simple online registration possible without having to load any special software. It has far less restrictions than other free providers and as far as I can tell the SMPT mailbox and web pages can be used from any dialup connection rather than from their domain - ideal for use overseas with a local ISP. The help is clear and concise and the rates seem as fast as one would expect from BT with its high speed backbone. This looks the first real competitor to Freeserve.
BTcellnet U.Genie SMS services:You do not have to have a BTcellnet phone to use the Genie ISP service but if you do then you can register to access extra sections of the U.Genie web site which are dedicated to SMS services. This gives an additional facility similar to Genie for email to your phone but without the redirection to email option so in that case Genie is the one to use. You can however send SMS messages to other phones on all 4 main networks using a web type email interface which has a simple address book. Again I can find no associated costs but they may be billed to the telephone itself. The registration process for the phone provides a password which is sent as an SMS message to the phone you register - a simple and effective security check.
Virus Checkers: I have downloaded the latest Virus checker with the latest Scan Engine (version 4) from the McAfee site after a series of new alerts. This was a massive 8 Mbyte download and within weeks it needed another update at several Mbytes to cope with the new Trojans. At least the latest version of McAfee covers downloads and email attachments automatically. Both Freeserve and Genie seem perfectly reliable for downloads taking up to an hour going on in the background and IE5 seems to allow one to restart a download if it has been interrupted.
CompuServe 2000: This looks like a disastrous mistake by CompuServe as it is largely incompatible and I can not see anyone changing all their email and web addresses and having to convert everything for minor enhancements. This means I have to seriously review staying with CompuServe for just the email address - the web site has already been moved to Freeserve due to the unreliability and poor service record of CompuServe.
Freezone: The Domain Name has been registered through Freezone, one of the free ISPs covered in my Guide to Selecting an ISP. They do a registration for £19.99 for a .co.uk or .org.uk domain and £49.99 for a .com, .net or .org. the registration last for two years and one also has to pay a £29.99 a year fee for domain maintenance ie. adding your domain to their Domain Name Server and redirecting services. The web space is pointed at their own free 20Mbytes space and the intention is to redirect the mail to BT Cellnet's U.Genie which has advantages when we are overseas. The redirection has not taken place yet and we are still picking up from the Freezone mailbox. The Freezone web space seems quite adequate. I did some tests by uploading to their web space and then accessing from various other ISPs - they are not quite as fast as Freeserve but I still got access via BT Cellnet's U.Genie, Freeserve and Freezone to be in the range of 2.5 - 2.6 Kbytes/sec on a Saturday at 1100 on a 100k zip file and a 28K baud modem - ie plenty good enough. Freeserve improves to 3.3 -3.4 kbytes/sec the second time you access showing their cache coming into play!
Managing the transition: We have now uploaded all the top level files to pcurtis.com and the folders containing newsletters and the two picture galleries. The OU pages are still mirrored at www.corinna1.freeserve.co.uk and will remain so until the end of the year. There are essentially 4 official entry points to our sites, Peter and Pauline Curtis's Information Service, Pauline's OU pages, The Howto technical Articles and Uniquely New Zealand and the challenge is to minimise the number of absolute links to the existing OU material on Freeserve and minimise the need for updating the mirror site. On other sites which are still used by some visitors such as CompuServe and Telinco we have replaced the four entry pages and the default by pages with notices about the move and absolute links so that the search engines catch up quickly. Automatic redirection breaks the Search Engine chain of links and increasingly causes your page to be unregistered. we again had terrible problems uploading to CompuServe and we will be glad when we can drop it for good - even email is becoming very slow to download and we took over seven attempts and over an hour, mostly online, to get a 460Kbyte attachment so as soon as the redirection situation is understood we will email everyone in our address books to inform them of the change.
PGP (Pretty Good Privacy) encryption: I therefore had a look at the PGP International Web site at http://www.pgpi.com and confirmed that Pretty Good Privacy (PGP) is an application and for secure e-mail and file encryption developed by Phil R. Zimmermann. Originally published as Freeware, the source code has always been available for public scrutiny. PGP uses a variety of algorithms, like IDEA, RSA, DSA, MD5, SHA-1 for providing encryption, authentication, message integrity, and key management. It now even has its own official protocol (RFC 1991).
PGP Secure "Virtual" Drives: PGP was initially written for secure communications, however my initial interest is in secure data. Fortunately the latest version not only gives high quality encryption for email but now also allows you to "mount" a completely secure virtual drive with 128 strong bit encryption using a passphrase (a passphrase is groups of words which are used in stead of a password for additional security). It can be set to demount if the machine is unused for a given number of minutes (typically 15 minutes) and when you turn off. Once the drive is mounted it seems indistinguishable from a normal drive (I have it set to Z:), you even have to format it before you can use it! I have put a link to the "Mounting and Unmounting" tool on the main Start menu - it could be in the startup group to force your hand if you plan to use the Secure drive a lot. You can have several secure virtual drives (called volumes) which can be individually mounted with different passphrases and several passphrases can be allocated to each drive. This gives total security against not only thieves but also the kids, the partner or, in the case of firms, staff accessing confidential information.
PGP Secure email: PGP use up to keys of up to 4096 bits and 128 bit strong encryption in various international standards.This level of encryption can not be broken even by a security agency and should be secure for several decades. There have been a lot of problems with this level of encryption being exported from the USA and eventually a loophole was found and each update has all the source code printed, carried out to Europe and scanned, all 6000 pages of it. The first time a team of 250 volunteers were used, it is now much more automatic. PGP has now been bought by Network Associates (who also own McAfee) but they still support free use of the "International" versions which are now up to 6.02i. PGP is integrated seamlessly into the clipboard and most standard email packages (other than Netscape) and operating systems. Under Windows 95/98 there is a new icon in the tooltray so you can also work on files or the clipboard (encrypt/decrypt and sign/verify and work with keys etc). There are also buttons on the Windows Explorer toolbars and entries on the drop down menus - they can also be added to most standard email packages and the clipboard can also be used.
What are Public and Private Keys? Secure email needs cooperation between the parties and the information needed to encode and decode are called Keys - the longer the keys the higher the security - PGP can use keys of up to 4096 bits. When you want to be able to receive secure email you give people copies of your Public Key which anybody sending email to you can use to encrypt it. The key does not enable anyone to decode it - only you have the Private Key used for decryption. When you want to send email you first have to get a copy of the recipients Public Key which he can email to you or may well have on his web site or a Key Server where you can search by name or email address. When you have collected it you can add it to your Key Ring and associate it with his email address. We have not added our Public Key to a Key Server yet - we will probably eventually add it to the PGP Key Server when we are sure we are stable. Creating the keys was easy and we have made a backup to a floppy which will live in the Bank strong box. When I exported my Public Key it looked just like an ascii file full of random characters and has a .asc file type which is now associated with PGP - a recipient running PGP just clicks on it and it asks which Key Ring to add it to.
PGP Documentation and Installation: This can only be an overview of a very powerful utility which seems to have been thought through. There are many additional things you can do including deleting files so they can not be recovered and clearing all the unused disk space of fragments of information. I do not have the time to cover all these or go into the background so it is well worth having a look at PGP International where there are series of manuals and background documentation - they certainly convinced me to try it out. It is circa a 5.5 Mbyte download with another 1.5Mbytes of interesting Adobe Acrobat (.pdf) files. These are also in the main download. It seems to occupy about 10 Mbytes of disk space when installed.
Adobe Acrobat version 4 I also downloaded the latest version of Acrobat and installed it on the "trial" drive. I uses a large amount of extra space with little apparent benefit unless you need to be able to run java programs in the .pdf files. it is true that the interface is a little more refined and it is easier to work through a document. Eventually I am sure I will have to upgrade on the main machine but at present I will live with the more compact version 3.0.
McAfee Securcast Backweb I downloaded the Backweb software so I can use the McAfee Securcast channel to download virus updates in the background. It seems to work well and uses the spare capacity on any open DUN. There were a lot of back copies of virus alerts and Engine and Data files initially after which it just produces the weekly updates and the occasional alert which Pops-up on the screen. The updates can be installed at a latter stage.
Patches downloaded and Installed:
Hypermart and Netmechanic tools Hypermart have links and partner programs which provide a lot of very useful Web Authoring tools. The most useful general utilities are actually those of Netmechanic and they can also be accessed directly at http://www.netmechanic.com. These include a Search Engine submission to 12 popular engines and a number of Utilities to check the site and your pages. The utilities I tried were the link checks, the HTML checks (you can chose the version), Browser compatibility checks and estimates of download times. The more complex tasks are done offline and when they are complete you are notified by an email which provides a web page address linking to the results which are held for 2 days. A very useful utility takes a .jpg file and shows you what happens if you reduce the memory size - each reduction swops back and forth to the original when you move the cursor over it so you can accurately choose how much degradation you can tolerate.
Forms: The major new thing I had to do was get forms working on the site. Unlike Freeserve there is no ready made script to take the POST from your form and email it to you - hypermart do however support PERL so you can get hold of one of the Freeware scripts to do it for you. A form is specified as part of an HTML document. The user fills in the form and, when complete, submits it by clicking a button. The Browser then sends the form's contents as designated in the HTML code, to the server. The HTML code that does this is all in the FORM element, The ACTION attribute is a URL specifying the location to which the contents of the form is submitted to elicit a response. The way data is submitted varies with the access protocol of the URL, and with the values of the METHOD attributes. The most common implementation is to use method=POST with action=URL where URL is the address where you have put a script file which contains PERL or similar code to read the data which has been POSTed to it, process it and send it out as an email. The way it is all handled is picked up by the script from a series of standard variables which you define in your HTML in addition to those that the user fills in.
What is a cgi-bin ? The above sounds fairly simple as is the case when all the hard work has been done for you and you have a framework to modify as on Freeserve. Anywhere else you need to find out a lot of information about the server and what it permits you to do. Firstly one must remember the Script file is a program which you are running on the server - this could run amok and take the server down so often such programs were only allowed to be run from within a single directory in a carefully controlled manner. program put into this directory were very carefully tested and normal users would not be allowed to load programs or they were very careful checked first. This directory was almost always called cgi-bin. Your server may contain scripts you are allowed to use in its cgi-bin. More recently the servers have allowed programs with extension .cgi to run in users directories so you may well find that if a PERL program (usually with extension .pe) is renamed to .cgi it can be run and that the server does sufficient runtime checking to avoid disasters - in the case of hypermart the errors are trapped and you can even look at the error log.
Locations of PERL and Mail programs: I tried to get scripts to run and found that it was not as easy as all this would indicate. Firstly the script needs to know where Perl is located on your server - Hypermart seems to be fairly standard and has the latest PERL 5.003 located at /usr/local/bin/perl and for compatibility PERL 4 is also available at /usr/bin/perl - one of those two is usually a good bet. You also need to know where the mail program is located and in the case of Hypermart they tell you it is at /var/qmail/bin/qmail-inject . Many servers use SendMail rather than qmail-inject and I do not know where that would normally live.
Permissions and chmod : The next problem is that one has to set up permissions on the cgi-bin scripts so they can be read and executed by yourself and other parts of the system. This is done on a typical Unix server by a program called chmod and you find references to chmoding to 775 etc. This strange number seems to be an octal number and refers to setting the read, write and execute bits for yourself, your group and the world so 777 sets everything and 775 would prohibit the world from writing your script. HTML documents and images are always set up for you but you do seem to have to set permissions for cgi scripts. Hypermart have utility on the site to do it otherwise WS_FTP can do it via a nice interface but only if you right click the file name you want to change - it does not appear on the menus. Take great care as you can remove your own permissions and then you can do nothing but get in touch with the web space provider and beg.
Uploading Perl scripts - use ASCII mode: I thought by now I should have it all working but not true. After many hours and surfing round various FAQs etc I discovered that you must upload PERL scripts in ASCII mode, if you do it in Binary which is what I do with everything else they do not work and come back with error 503. I was worried about editing the scripts but it seems that Write does no harm when editing text files and I could work on a .pe or .cgi file without problems. (Notepad has limitations on file size and is much more basic)
PERL scripts for utilities and testing. I have spoken about scripts for FORMs but one does not want to use that for testing and I discovered various PERL utilities such as counters and time displays as well as the very important FormMail at Matt's Script Archive at http://www.worldwidemart.com/ I also did a short online tutorial at User Active at http://www.useractive.com where it takes you through the basics including uploading and running a simple CGI script in PERL. I saved that away for future tests. It looks much like HTML and the output is inserted into the HTML stream returned from the server to the browser.
Executing a PERL program as a SSI: A common way to run a PERL program is as a Server Side Include (SSI) which few of the free ISPs support but is supported on the Hypermart web space that I am using for this job and I did my first tests using simple PERL scripts run as SSis. A typical way to do this is by having a statement like <!--#exec cgi="/cgi-bin/perltest.cgi"--> in your HTML page which asks the server to execute the cgi program perltest.cgi and puts it's output into the HTML stream sent to your browser when the page is accessed.
Secure Servers: Hypermart offer a Secure Server Interface for credit card etc transactions. It is very easy to use and one simply accesses the same pages you have uploaded but using the full address of https://serverxx.hypermart.net/yourname/yoursecurepage.htm instead of http://yourname.hypermart.net/yourpage.htm. They tell you when you register which server you are on. Relative addressing works after the first access but you should leave the secure area with a full http://...... URL after the transactions are complete as the secure area currently turns the banners off and you need a tidy return. Accessing a Secure Site requires that your browser has the matching Certificates to the server and that they are in date. Hypermart uses certificates from Thawte which are supported with in date certificates by most browsers. Users of second and some third generation browsers may need to get updated certificates and Thawte provide a clever Graphic Link which checks your browser and changes the image it returns to tell you if your browser is OK or what to do. I have not been able to check it fully as I only have recent browser but I have put it below in case anyone is interested. They also provide a version which redirects to your secure page if everything is OK.