Showing posts with label Windows. Show all posts
Showing posts with label Windows. Show all posts

Monday, 30 April 2012

Moving Mail to a New Laptop

My wife was getting fed up with her eMac running slower and slower, plus incompatibility between her MS Office X and the MS Office 2010 in use at her company, so for her birthday she got a Windows 7 laptop. Until the weekend just gone, however, she didn't make much use of it because it was beyond her ability to move her mail account from Entourage to Outlook and to move all her files across. The address book was a particular problem.

Eventually we found ways to achieve almost all these migrations. The document files were the easiest part - local area network connection, turn on Windows sharing on the Mac, just drag the files over (we had to retry a few times because the connection reset while copying long files).

To migrate the mail messages, we decided to use Thunderbird instead of Outlook (this PC doesn't connect to a corporate Exchange server). Drag each folder from Entourage to a Finder folder or the desktop. This exports the messages in the form of a .mbox file. Rename to remove the extension and copy them to Thunderbird's Mail folder in the current profile.

The mail addresses turned out to be rather a tougher proposition. You can export each address card to a .vcf file, but Thunderbird found no useful information in them. It dawned on me later that this might be a line-ending issue - we could have run mac2unix and then unix2dos before importing. But by then we had solved the problem in a different way.

Entourage allows you to drag address cards to the Mac's native address book application. This was synchronized with Plaxo using my wife's account. Plaxo supports export of the address book in LDIF format, which Thunderbird was able to import without problems. If you can't use this route, you can also export the entire Mac address book in a single operation to .vcf (or possibly even to LDIF). I suggest adjusting the line endings to Windows conventions before attempting to import to Thunderbird.

Then it turned out that the vast majority of e-mail addresses that she needed were not in either address book. Entourage just maintains a cache of recently used addresses and we could find no way to export all of these into any sort of useful file format. To the rescue came Adam Haeder with a simple shell script to extract plausible e-mail addresses from mbox files. I was able to adapt it for use under Cygwin and enhance it a bit to cope with non-US addresses and escaped newlines and spaces. The resulting tab-delimited file was suitable for import to Thunderbird.

 #!/bin/bash  
   
 # This script will parse an mbox file, displaying all of the From: email addresses, removing ones that  
 # are from postmaster, mail admins, etc  
   
 FILE=$1  
   
 if [ ! -r $FILE ]; then  
  if [ -r /var/spool/mail/$FILE ]; then  
  FILE="/var/spool/mail/$FILE"  
  else  
  echo "Sorry! Neither $FILE nor /var/spool/mail/$FILE exists, or I can't read them"  
  exit  
  fi  
 fi  
   
 # Using cat to read the input means you can run this over many mbox files simultaneously  
 # dos2unix and mac2unix standardise line endings to LF only  
 # First sed script joins lines that have been split with a "=" at line end  
 # grep isolates lines starting with "From:" and egrep -vi rejects all non-human addresses  
 # Next grep discards any lines that contain no email address (@)  
 # The next sed script turns =20 into spaces and discards trailing spaces and "From:"  
 # The next converts [mailto:x@y.z] to <x@y.z> form  
 # Any bare SMTP addresses are converted to "x@y.z <x@y.z>"  
 # Any SMTP addresses with no friendly name but in "<>" delimiters are converted similarly  
 # Finally the "<>" delimiters are removed and replaced with a tab separator  
 # Result is sorted uniquely (ignore case & leading blanks) and converted to DOS line endings  
 cat $FILE | dos2unix.exe | mac2unix.exe |\  
 sed '/=$/{N;s/=\n//}' |\  
 grep "^From:" | egrep -vi \  
 "(postoffice|\  
 postman|\  
 administrator|\  
 bounce|\  
 MAILER-DAEMON|\  
 postmaster|\  
 Mail Administrator|\  
 Auto-reply|\  
 out of office|\  
 Mail Delivery System|\  
 Email Engine|\  
 Mail Delivery Subsystem|\  
 Mail.Administrator|\  
 non.deliverable)" |\  
 grep '@' |\  
 sed 's/=20/ /g;s/\s*$//;s/^From:\s*//' |\  
 sed 's/\[mailto:\(.*\)\]/<\1>/g;s/"//g' |\  
 sed '/^[^<]*$/s/^\(.*\)$/\1 <\1>/' |\  
 sed 's/^<\(.*\)>$/\1 <\1>/' |\  
 sed 's/^\(.*\S\)\s*<\(.*\)>/\1\t\2/' |\  
 sort -ubf | unix2dos.exe  

Friday, 14 January 2011

Oracle 10g problem under Windows Server 2003

It cost me the best part of two days' detective work to fix a problem with the Oracle DB Control service. Symptoms include:
  • "CONFIG: Waiting for service 'OracleDBConsoleorcl' to fully start" in cfgtoollogs\dbca\orcl\cloneDBCreation.log
  • "The oracleDBconsoleorcl service could not be started"
  • "[Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ias.ias.IASIntegration" in emoms.log
This started to happen after a failed import using the data pump (impdp) trashed the database and I rashly thought that I might clear up the problem by uninstalling and reinstalling Oracle 10.2.0.4.

Background: For a client project, we needed a local copy of the client's schema and stored procedures. I found it remarkably straightforward to install the Oracle product and a default database instance named orcl. However, attempting to import the client's exported schema and stored procs caused a rash of errors and then the SYSTEM user's password had been corrupted too. My suggestion: use the deprecated exp and imp tools instead. They're much more straightforward to use, AND the wonderful DDL Wizard is able to convert the exported file to DDL and PL/SQL that you can run in SQLPlus instead of overwriting your database contents willy-nilly.

Solution: I wish there was one. The problem appears to occur in the final phase of the database configuration assistant - after creating and configuring the database it tries to start up the console service for it and fails, for as-yet unfathomable reasons. Could it be to do with the fact that we also run MS SQL Server on the same box? Yet this was fine before I tried to reinstall Oracle. (As I eventually learned, I need not have redone the entire installation, because the database configuration assistant is quite good at deleting the database instance and creating a new one - in fact, this is what the installer invokes anyway).

Luckily the whole thing runs on a virtual machine, so I am going to get the previous week's VM backup image restored. Luckily we hadn't been doing a lot of work on that machine in the meantime.

My grateful thanks to the many bloggers who came up with possible solutions - none of which worked for me:


Monday, 2 August 2010

Migrating a Subversion repository

In my current project, there are a number of development teams whose code repositories have grown, separately or together, in a somewhat haphazard fashion. As part of the initiative to introduce a more streamlined development process, therefore, I want to host all repositories on one server and reorganise them in a more uniform and logical way.

Using Subversion, it is relatively straightforward to move whole repositories, but taking one branch out of an existing one to add to a new repository (without losing the change history!) turns out to be surprisingly hard. After much experimentation, the best way turns out to be via Mercurial's excellent "convert" extension. Unfortunately, conversion from Hg to Subversion simply fails under Windows, so you need to use Linux.

This particular customer has a totally Windows-based infrastructure, as well as a particularly draconian firewall that uses Windows authentication exclusively. The following recipe therefore had to take that into account. Names have been blanked out where necessary to preserve client confidentiality.

  1. Install the VMWare player and a Linux appliance, such as the Ubuntu 10.04 LTS image with VMware Tools.

  2. Lightly edit the VMWare virtual machine configuration file (Ubuntu.vmx) to ensure that it has at least 1024MB of RAM. Start up the Linux VM and upgrade existing packages. This turns out to be a little more difficult than you might expect, because the Synaptic package manager (unlike Firefox) is unable to pass your Windows username and password as NTLM credentials to the network proxy server. Therefore you have to proceed as follows:

    1. Make sure that the VMWare player's network connection is set to "NAT"

    2. In System -> Preferences -> Network Proxy, set the manual HTTP proxy to the server and and port that your Windows web browser uses. If the Web browser's Internet settings nominate a proxy auto-configuration script, you may be able to use that instead. Don't forget to set your Windows username and password. Apply system-wide (you will have to supply your Linux user password twice).

    3. Open Firefox and navigate to http://sourceforge.net/projects/cntlm/ - download the .deb package and allow the package installer to install it. This is a local proxy server that logs onto the firewall on your behalf.

    4. Now configure the cntlm service as follows:
      • As superuser, edit /etc/cntlm.conf and set the following values (see the documentation):
        • Username (your Windows user ID)
        • Domain (your Windows domain)
        • Workstation (your PC name - not the full FQDN)
        • Proxy (proxy FQDN:proxy port)
        • Proxy (backup proxy FQDN:proxy port)
        • Listen 3128
      • Make sure that the permissions on cntlm.conf are -rw-r--r--
      • As a normal user, run the following command:
        cntlm -v -I -M http://test.com
      • Copy and paste the resulting profile (the lines between -------------- markers) into the cntlm.conf file - search for "#Auth" to find the insertion point
      • As the superuser, start up the cntlm service:
        /etc/init.d/cntlm start

    5. In System -> Preferences -> Network Proxy, set the manual HTTP and FTP proxy servers to localhost and port 3128. Username and password should still be your Windows credentials. Apply system-wide as before. Shutdown Firefox and open it again to check that the proxy settings are correct.

    6. Open System -> Administration -> Synaptic Package Manager (it will request your Linux user password). Go to Settings and configure a manual proxy just as in System Preferences.

    7. Still in the package manager, click Reload. You should see the package files being downloaded successfully. Click "Mark All Upgrades".

    8. Still in the package manager, type "subversion" into the search box. Mark both "subversion" and its dependencies and "python-subversion" for installation. Similarly for "mercurial". Finally, click "apply", click the Apply button, and sit back.

    9. Part way through the installation process, you may encounter a warning that the Linux system image cannot be safely installed without GRUB. However, the grub loader has been superseded by grub-pc, so you can safely ignore this (click the "go ahead without grub" checkbox).

  3. You can also upgrade the VMWare tools, which the player will claim are out of date. This is of unproven usefulness. It doesn't take terribly long, so you may as well. After downloading the upgrade, VMWare Player mounts it as a CD image under /media (run "df" to find out what it's called). The trick is to unzip from there into /tmp or your home directory, because the mounted volume is read-only. Then run "sudo perl <install-script>" to upgrade the software.

  4. Assuming all has gone well, you should now be able to run Mercurial and Subversion from the command line. Test it by running the following commands:
    hg help
    svn help
    svnadmin help

  5. You need to cache the subversion authentication parameters in order that Mercurial can invoke subversion non-interactively. To do this, make sure you are logged in as an ordinary user (not root) and run:
    svn log -l 2 REPOSITORY_URL
    where REPOSITORY_URL is the address of the repository from which you wish to migrate a branch.
    When prompted for the "user" password, just hit return. Now enter the username of a real user with access to the target repository, and hit RETURN. Next enter that user's password and hit RETURN. You should be rewarded with two lines of history from that repository.

  6. You must enable the convert extension by uncommenting the line containing "hgext.convert =" in /etc/mercurial/hgrc.d/hgext.rc.

  7. Now you're all set to do the export. Create a folder (e.g. "repo_migration") and change directory into it. Create a file-mapping file such as myproject_mapping.txt:

    include branches/Development/myproject
    rename branches/Development/myproject trunk

    See the hg convert documentation for details of the mapping format.

  8. The following command does the business. It will probably process around 100 revisions per minute on average, so you've got time to make lots of cups of tea:
    - hg convert -d svn --filemap myproject_mapping.txt -s svn REPOSITORY_URL NEW_REPOSITORY
    where REPOSITORY_URL is the address of the repository from which you are migrating, and NEW_REPOSITORY is the name of the local Subversion repository you want to create as the result.

  9. If the conversion starts to slow down, you can open a second terminal and, as superuser, set the nice level of the running hg process to -20, which should increase its priority to maximum. I'm not sure how much this helps, since the process seems mainly network-bound.

  10. Finally, copy the new local repository to a location from which your Subversion server can serve it, and Bob's your parent's brother.

It has just occurred to me that the reason this doesn't work under Windows is probably because we didn't have the python-subversion package installed. However, that is just a guess because the diagnostics given (exit code 1) provided no clue.

Ideally of course, you should just export the existing repository to Mercurial and let everyone use that instead of Subversion from now on. Unfortunately some tools we want to use, such as JIRA are not yet Mercurial-aware. Besides, we don't want to incur the overhead of training developers to work with a new SCM tool - it isn't that long since they even began to use Subversion.

Friday, 23 July 2010

Turn any program into a Windows service

Today, Oisin Mulvihill introduced me to his company's open source product, Service Station. It's very simple, very effective and stunningly useful. I am really surprised that it isn't much better known. With one short shell command, you can install any script or application (headless, command-line or GUI) as a Windows service that can be started, stopped and managed through the Services control panel. As a bonus, Service Station restarts programs that have crashed, cleans up all spawned processes when stopping a service, and provides comprehensive logging via the Windows Event Viewer - it really could not be simpler for the system administrator. It is suitable for every version of Windows from 2000 onwards.