Munging PDF files and pages

Having scanned over a dozen handwritten pages from my notebook into a PDF file, I had to invert every other page to make it possible to read the whole thing. It took me a while to find the answer, but YOU can go straight to the solution!

The command line is

pdftk.exe scannedfile.pdf cat 1 2S 3 4S 5 6S 7 8S 9 10S 11 12S output readablefile.pdf

Ramaze - first impressions continued

My earlier posting was getting too long, so I'm continuing in this new one. Meanwhile, the tutorial's author has kindly left a comment on the original posting, indicating that the tutorial is due to be updated pretty soon. I look forward to that!

Chapter 11 of the tutorial can be used pretty much as-is, but I found that after an error, the redirection specified by the error method would be overwritten by the helper aspect. After some research, I discovered that the latest version of Ramaze supports a redirection status operator, redirected?. Using this, the helper aspect can be written very simply:

helper :aspect
after( :create, :delete, :open, :close ) {
  redirect Rs() unless redirected?
}

Chapter 12 of the tutorial needs no changes, except that you don't have to add the flash section to the page template because we did it earlier.

Now I wanted to convert the application to something more suited to enterprise deployment (and also more suited to shared hosting deployment - typically such environments provide an Apache server and MySQL database). So the first thing to do was to move from YAML to SQL - I chose MySQL.

If you haven't yet installed MySQL, do so before the next step. Take care to ensure that your PATH environment variable contains the MySQL binary as well as its libraries (hold down the Windows key and press PAUSE to bring up the system properties, then choose Environment Variables in the Advanced tab). For example, on my system the System PATH begins:

C:\ruby\bin;
C:\Program Files\MySQL\MySQL Server 5.0\bin;
C:\Program Files\MySQL\MySQL Server 5.0\lib\opt;
C:\Program Files\MySQL\MySQL Server 5.0\lib\debug;
...

Add a new database for the application and create a user account named "ramaze" with password "TodoList" for both localhost and remote-host access:

mysql -u root -p
******
CREATE DATABASE IF NOT EXISTS todolist_db;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE,
  DROP, RELOAD, PROCESS, FILE, REFERENCES, INDEX,
  ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
  LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW,
  CREATE ROUTINE, ALTER ROUTINE ON *.*
  TO 'ramaze'@'localhost'
  IDENTIFIED BY 'TodoList' WITH GRANT OPTION;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE,
  DROP, RELOAD, PROCESS, FILE, REFERENCES, INDEX,
  ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
  LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW,
  CREATE ROUTINE, ALTER ROUTINE ON *.*
  TO 'ramaze'@'%'
  IDENTIFIED BY 'TodoList' WITH GRANT OPTION;
quit

I decided to use Sequel as my database access layer. You probably also have to install the gems mysql and sequel before the code will work - I am not sure about the mysql gem, as it was one of the things I installed before I eventually discovered the need to set up the PATH correctly.

gem install sequel
gem install mysql

I tried choosing the 'ruby' option, but this was unable to generate the native code on my machine. So I uninstalled it and tried again, choosing the 'mswin32' option, which worked fine.

Now take a safety copy of your model file, todolist.rb, and start modifying it to use MySQL instead of YAML. Replace the following lines:

require 'ramaze/store/default'
TodoList = Ramaze::Store::Default.new('todolist.yaml')

with the following lines:

require 'rubygems'
require 'sequel'

DB = Sequel.mysql('todolist_db',
  :user => 'ramaze',
  :password => 'TodoList',
  :host => 'localhost')

class TodoList < Sequel::Model(:tasks)
  set_schema do
    primary_key :id
    varchar :title, unique => true, :null => false
    boolean :done
  end
end

To begin with, I tried using the title as the primary key. But I soon found that not only was it necessary to define the title field first and then separately to name it as the primary key, the user-supplied title was not always suitable as a key value due to the presence of shell metacharacters and so on. So I decided to go with the Sequel flow and use a system-generated ID as the primary key.

Next I found that the model didn't support the method original(), which the original simple model supports. So I supplied one myself (subsequently found to be unnecessary after refactoring main.rb):

  # Copy all records into a list
  def self.original
    tasks = []
    self.dataset.each {|r| tasks.push [r[:title], {:done => r[:done]}]}
    return tasks
  end

More important was to add some initialisation code after the class was defined:

unless TodoList.table_exists?
  DB.transaction do
    puts "Creating table 'tasks'\n"
    TodoList.create_table
  end
end

Now try running the application. It seems to work, but nothing gets stored in the database. At this point we have to bite the bullet and refactor the main module to support a more relational view of the underlying data model.

In the index() method, we'll start using the ID of each task to identify it instead of the title. Instead of requesting an array of key-value pairs from the original() method, we get an array of task objects from the dataset() method. That of course implies that we have to extract the title and the ID from the task object and use them as appropriate:

  def index
    @title = ["To-Do List"]
    @tasks = []
    TodoList.dataset.each do |task|
      id = task[:id]
      title = task[:title]
      if task[:done]
        status = 'done'
        toggle = A('Open Task', :href => Rs(:open, id))
      else
        status = 'not done'
        toggle = A('Close Task', :href => Rs(:close, id))
      end
      delete = A('Delete', :href => Rs(:delete, id))
      @tasks << [title, status, toggle, delete]
    end
    @tasks.sort!
  end

The methods open(), close(), task_status() and delete() have to change because they all now take an id as parameter:

  def delete id
    unless TodoList.delete id
      failed "Cannot delete task no.: #{id}"
    end
  end

  def open id
    task_status id, false
  end
  
  def close id
    task_status id, true
  end

  def task_status id, status
    unless task = TodoList[id]
      failed "No such task no.: #{id}"
      redirect_referer
    end
    task[:done] = status
    TodoList[id] = task
  end

I decided to override the []= method of the model, so that tasks would actually be written to the database. It was an easy step from there to create a new row in the tasks table whenever the ID parameter to this method was absent or nil. So the create() method becomes:

  def create
    if title = request['title']
      title.strip!
      if title.empty?
        failed("Please enter a title")
        redirect '/new'
      end
      if TodoList.find(:title => title)
        failed("Task '#{title}' already exists")
      else
        TodoList[nil] = {:title => title, :done => false}
      end
    end
  end

Note the check for duplicates, which is easy to do now that we can look up titles in the dataset.

Turning to the file todolist.rb, here are the methods I had to add to allow tasks to be added and deleted in the database:

  def self.delete(id)
    puts "Attempting to delete '#{id}'\n"
    DB.transaction do
      if task = TodoList.find(:id => id)
        task.destroy()
      else
        puts "Not found\n"
        return false
      end
    end
  end

  # Assignment should update the underlying database
  def self.[]=(id, values)
    DB.transaction do
      if (id == nil || !(task = TodoList.find(:id => id)))
        task = TodoList.new
      end
      task.title = values[:title]
      task.done = values[:done]
      task.save
    end
  end

That's pretty much it. But before I stopped, I wanted to prettify the user interface a bit. I didn't like the fact that the column widths tended to change whenever the sole "not done" item was added to or deleted from the list, and in any case I preferred clickable icons to text links. So I designed some icons:

not done

done

delete

Feel free to copy the icon images. To let the Mongrel server access these, they have to be placed in the folder "public" of your project. Then I redesigned the index() method and the corresponding index.html file very slightly to use these (first declaring some constants):

  DELETE_ICON  = '<img src="delete_sml.gif">'
  NOTDONE_ICON = '<img src="notdone_sml.gif">'
  DONE_ICON    = '<img src="done_sml.gif">'
  
  # the index action is called automatically when no other action is specified
  def index
    @title = ["To-Do List"]
    @tasks = []
    TodoList.dataset.each do |task|
      id = task[:id]
      title = task[:title]
      if task[:done]
        toggle = A(DONE_ICON, :href => Rs(:open, id))
      else
        toggle = A(NOTDONE_ICON, :href => Rs(:close, id))
      end
      delete = A(DELETE_ICON, :href => Rs(:delete, id))
      @tasks << [title, toggle, delete]
    end
    @tasks.sort!
  end

    <p><a href="/new">New Task</a></p>
    <?r if @tasks.empty? ?>
      <p>No Tasks</p>
    <?r else ?>
      <table>
  <?r @tasks.each do |title, toggle, delete| ?>
    <tr>
      <td class="title" > #{title}  </td>
      <td class="toggle"> #{toggle} </td>
      <td class="delete"> #{delete} </td>
    </tr>
  <?r end ?>
      </table>
    <?r end ?>

The full source code is attached in the comments to this post.

Headlights

While I'm on the subject of motoring, why is it that some people insist on driving along well-lit roads at night with headlights on full-power dipped beam or even high beam? It doesn't allow them to see any better, but significantly impairs the ability of oncoming drivers to see what's near their car. Many's the time when I have been uncertain of the amount of space between a dazzling pair of headlamps on the opposite side of the road and parked vehicles on my side, so that I have had to stop. Once I nearly knocked down a guy crossing the road behind a car coming towards me - he was wearing dark clothes and was simply hidden by the glare from the headlamps.

The only situation in which I will agree that the use of something stronger than sidelights is useful is when I'm approaching a corner. Traffic coming from another direction announces itself by the headlight beam cast ahead. Yet even this is sometimes misleading, as there are plenty of idiots who will park at the side of the road leaving their headlamps switched on. As we don't need this kind of advance warning during the daytime, why should it be required at night?

I always choose sidelights or dim-dip headlights when motoring in areas with good street lighting, out of consideration to other drivers - yet they frequently flash me, assuming that I have forgotten to switch on my lights properly. Other sins include the use of fog lights front and rear when there is no fog. This is particularly dangerous when the road is wet, because they are mounted low down and the reflections off the water can almost blind other road users.

Rules 113 and 114 of the Highway Code sum up the requirements of the law, which seem perfectly common-sense:

You MUST

• ensure all sidelights and rear registration plate lights are lit between sunset and sunrise
  
• use headlights at night, except on a road which has lit street lighting. These roads are generally restricted to a speed limit of 30 mph (48 km/h) unless otherwise specified
  
• use headlights when visibility is seriously reduced (see Rule 226)
  

Night (the hours of darkness) is defined as the period between half an hour after sunset and half an hour before sunrise.

You MUST NOT

• use any lights in a way which would dazzle or cause discomfort to other road users, including pedestrians, cyclists and horse riders
  
• use front or rear fog lights unless visibility is seriously reduced. You MUST switch them off when visibility improves to avoid dazzling other road users (see Rule 226)
  

In stationary queues of traffic, drivers should apply the parking brake and, once the following traffic has stopped, take their foot off the footbrake to deactivate the vehicle brake lights. This will minimise glare to road users behind until the traffic moves again.

Driving and Parking in London

If you want to keep your sanity, don't own a car in London. Occasionally however, you need one. For those occasions, I'm very much in favour of car-club schemes like CityCarClub, ZipCar and StreetCar, but none of them has a depot anywhere near my house yet. So I still own a car, but I don't use it very much. At certain times of day, I have found that it actually took me longer to drive somewhere than it would have taken to walk. And you don't have to pay a congestion charge for walking.

However, I'm currently on a project that involves flying to a customer site in Europe several times a month, leaving London City Airport very early in the morning - so early in fact that I would miss the flight if I tried to go by public transport. In the past, I have used a minicab instead, but fares have recently gone up so much that I resorted to driving to the airport myself. If you stay at least two days, you can book a space at the airport car park on the Internet and save a fair bit of money, but not for a single day - and the parking costs almost as much as the minicab.

To the rescue came a new service called ParkAtMyHouse. Such a simple yet elegant idea - anyone with a garage or a bit of parking space can advertise it on-line and you simply haggle about the price. The web site handles the reservation and communication between owner and renter of the space. There's a feedback mechanism similar to eBay. The service levies a fee of 10% of the rent paid for its services, which seems reasonable as you would not have any other way of discovering these cheap parking spaces. You can rent by the hour, day, week, month or longer. I didn't find anything very close to the airport, but only two stops away on the DLR was someone willing to rent me a space for only £5 a day - job done!

Ramaze - first impressions

I've spent the past couple of days evaluating Ramaze. I started with the tutorial To-Do-List example and tried to move beyond that to something resembling a real web application backed with a proper database (MySQL). In general, I feel that this framework is more intuitive to use and therefore gets results faster than e.g. Ruby on Rails, if you're having to learn both from scratch. I also found it very easy to refactor the code, because it encourages a fairly aspect-oriented style and all changes are loaded immediately without having to re-start the server. Last but not least, the designers of Ramaze have considered enterprise-level deployment requirements from the very start, so your Ramaze application can run on anything from a shared virtual host rented for a few dollars per year to an enterprise server farm or on-demand compute-cloud configurations.

Ramaze also leverages the power of Ruby in very elegant ways. For example, here's a description of a basic content-management system based on Ramaze in just a single page of code.

I'm going to walk you through the to-do list tutorial, which dates back to 31st January 2008. Ramaze has evolved a fair bit since then and I have had to find out what the changes are to adapt the tutorial to the current version. Hopefully my observations will be incorporated in a future version of the tutorial. All of my work was done on Windows XP (blame my employer, not me :-) so the installation instructions etc. are related to that.

First of all, if you don't yet have Ruby and Ramaze installed, here's how you do it:

• Grab the latest Ruby installer
• Install with default options
• Open a command shell and make sure that your Ruby bin directory is on the PATH
• gem install ramaze

You may find yourself being asked to approve the installation of various dependencies. Just agree to each one. If gem offers you options (ruby, java, mswin32 etc.), choose "ruby" or "mswin32" (assuming you're on a Windows 32-bit machine).

My next big problem came with Eclipse. I wanted to use Eclipse as my IDE for Ruby and Ramaze. Trying to use the Help -> Software Updates feature to install Ruby support, I discovered by much trial and error that it was possible to do so, provided you configure the update site http://update.aptana.com/update/rails/3.2/ and select "Uncategorized -> Ruby Development Tools" and nothing else. Using this plug-in, you get a language-sensitive editor and the ability to launch Ruby applications, not much else, but it's a whole lot better than using just a text editor. The first time you try to open the Ruby perspective, the Hierarchy pane displays an error message ("Could not create the view: org/eclipse/search/internal/ui/text/ResourceTransferDragAdapter"). Just close it and use the Navigator pane instead. I'd be interested to know if there is a better Ruby plug-in for Eclipse.

While I was about it, I also installed the Web Page Editor from one of the Eclipse update sites. This came in useful when I was working on template pages later.

Then I started working my way through the tutorial. The first discrepancy came in chapter 3 - the View. Ramaze has evolved so that you supply just partial pages under the view folder. So the view/index.xhtml file in the tutorial should contain only the part of the page that will be supplied to the main page template in the variable @content. It looks like this:

  <ul>
   <?r
     TodoList.each do |title, value|
     status = value[:done] ? 'done' : 'not done'
   ?>
     <li>#{title}: #{status}</li>
   <?r end ?>
 </ul>

Of course, at this stage you won't have assigned a value to @title, so your page will be rendered without a page title or first heading.

Also on this page, ignore the instruction to "run ramaze" from the command line. That won't work under Windows. Instead, Eclipse comes to the rescue. Just right-click the start.rb file in the Navigator pane and select "Run As... -> Ruby Application". When you want to terminate the server, click the red stop button in the console pane. Subsequently you can re-run the server by selecting start.rb from the pull-down menu under Eclipse's green Run button.

In chapter 4 of the tutorial, again please omit the HTML scaffolding and just write

  <?r if @tasks.empty? ?>
   No Tasks
 <?r else ?>
   <ul>
     <?r @tasks.each do |title, status| ?>
       <li>#{title}: #{status}</li>
     <?r end ?>
   </ul>
 <?r end ?>

Now add the following line at the very top of the index method:

  @title = ["To-Do List"]

The standard page template inserts this value both in the page title and the top-level heading. You may have noticed that the MainController contains the line

  layout '/page'

This indicates that the file view/page.xhtml will be used as the template, not src/element/page.rb as described in the tutorial.

In chapter 5 of the tutorial, obviously you should not add the level-one heading as shown. Just put the link at the very top of the file view/index.xhtml.

Next, the new file view/new.xhtml is also much simplified:

  <a href="/">Back to TodoList</a>
 <form method="POST" action="create">
   Task: <input type="text" name="title" /><br />
   <input type="submit" />
 </form>

Before creating an action for create, as described in the tutorial, you will also need an action for new. This is the only way that I have found to generate a title for the new.xhtml page. Insert the following in main.rb:

def new
@title = ["Create a new To-Do List item"]
# See view/new.xhtml
end

Chapter 6 of the tutorial remains pretty much unchanged. Just remember not to delete the @title initialisation at the very top of the index method.

In chapter 7 of the tutorial, I found that I was able to add and delete tasks to my heart's content until I tried deleting a task such as "Lunch with Bob?". Ramaze has a very simple mapping scheme from URL path to action argument, which stops parsing the path_info as soon as it encounters a "?" or ";". I spent a considerable amount of time trying to think of ways to circumvent this and pass the offending characters as url-encoded escape sequences, but in the end it was all to no avail. The lesson is that you would be well advised to never use user-supplied values as keys into your data. Instead, treat the user-supplied string as a data field and generate your own system ID for each item. By the way, this lesson applies equally well in other frameworks. Following my completion of the tutorial I will show how I refactored the code to achieve greater robustness. For now, just avoid using metacharacters in your task titles!

The solution to the problem of stuck tasks was of course to edit the YAML database file to delete the offending characters.

Chapter 8 of the tutorial seems to me now to be more or less redundant. All I had to do was to modify the file view/page.xhtml slightly to incorporate the error message from the flash object. Here's the resulting body section:

   <h1>#{@title}</h1>
  <div id="error">
    <p>#{flash[:error]}</p>
  </div>
  <div id="content">
    #@content
  </div>

Note that there is no need for any logic to decide whether an error message is present. If it's there, it gets rendered as part of the page. Otherwise the renderer omits that element.

The prettification step in chapter 9 of the tutorial still applies as it stands, except that the stylesheet entries are inserted into the file view/page.xhtml instead of the Page element. I also added a style for the error message, where present. The resulting HTML header now contains the following entries:

   <style type="text/css">
    body      { margin:      2em; font-family:Verdana }
    #content  { margin-left: 2em;               }
    #error    { margin-left: 2em; color:red;    }
  </style>
  <style type="text/css">
    table     { width:       80%;               }
    tr        { background:  #efe; width:100%;  }
    tr:hover  { background:  #dfd;              }
    td.title  { font-weight: bold; width: 60%;  }
    td.status { margin:      1em;               }
    a         { color:       #3a3;              }
  </style>

I had real fun in chapter 10 of the tutorial. Firstly, I had to install the mongrel adapter:

gem install mongrel

For both mongrel itself and its fastthread dependency, I chose the "ruby" option.
When I tried to start Ramaze again, it complained that some other process was already listening on port 80. After some detective work with process explorer, I discovered that this was a little peer-to-peer application called kservice from Kontiki Inc. I had probably unwittingly installed it with some Internet media player. After following the uninstall instructions kindly blogged by "Geoff" I was finally able to start my server on port 80.

(to be continued)