2010-11-30

Logging, It's a Good Thing

If Martha Stewart did computer programming, she'd say "Logging, it's a good thing.". Unfortunately, many programmers including me do not make a habit of it.

There are different reasons to do logging:

  • Help figure out what is going on during programming or debugging.
  • Track normal operations of the app.
  • Report trouble or errors.
Yohan Liyanage wrote a nice blog post about how to log. In particular he includes a table with clear descriptions of each of the six levels of logging as specified by the Apache Commons Logging project:
  • TRACE
  • DEBUG
  • INFO
  • WARN
  • ERROR
  • FATAL
Basically, the first two are for programming & debugging. The middle one, INFO, is for tracking normal operations. The last three are for recording problems.

I ported that "Log" interface to REALbasic, and offer it to everybody. By using an interface, you can switch logging implementations, at design time or run time, without changing the countless logging calls littered throughout your app's code. Also, if everyone in your team, or even the entire REALbasic community, used the very same interface it would be easier to share code containing logging calls.

Robert Elliot wrote a another interesting logging blog post

Aaron Ballman in his "Ramblings" book explains how useful logging is for debugging a GUI app. When running your app from the REAL Studio app, using the debugger can affect the state of your app's GUI. This is one instance of the Observer Effect. This happens to be why he invented the Remote Debugger Stub feature of REAL Studio 2010. The common misconception is that the purpose of Remote Debugger Stub is to test across platforms. But actually it came from his own frustration with this observer effect. By using the Stub, even on the some platform, you can debug without affecting the state of your app.

2010-11-22

"Shared Memory" settings for Postgres on Mac OS X

Shared Memory Properties
Property Description Reasonable Value Snow Leopard Default
xxx yyy zzz nnn
xxx yyy zzz nnn
xxx yyy zzz nnn

The best I can figure out: Apparently "shared memory" is a way for separate processes to share information. Since Postgres launches several processes for itself, and then one more process for each connected user, the processes must have access to "shared memory" in common. Setting this amount is critical to successful use of Postgres. The bulk of the database operations take place in that memory. So the bigger the database, the more memory you need allocated to "shared memory".

This is not normal among Mac apps. Most Mac apps ask for more memory than they need. Mac OS X responds to the app by lying, saying "Yes, you can have that much memory". But in fact the app gets a mere fraction of that. When the app starts using that memory, and runs low, the virtual memory system allocates more memory as actually needed. A few apps, such as 4D or Photoshop, grab big chunks of real memory to be managed internally by that app rather than be managed by the Mac OS X virtual memory system.

Rather than go either of these routes, Postgres and other SQL database engines (DB2, Sybase, etc.) use "shared memory". The catch is that by default Mac OS X may have set low limits to the amount of shared memory allowed.

So how do we view the current shared memory settings and set them in Mac OS X? The answer: In the command line, using a tool called sysctl.

Check your settings by typing into your Mac's Terminal program:
sysctl -a
We can use a shorter list, as we don't care about most of those settings. Typing a partial name, up to any of the periods, acts as a filter. Type:
sysctl kern.sysv.
By the way, "kern" is usually short for "kernel" meaning the heart of the operating system.

Postgres also cares about:
  • kern.maxprocperuid
  • kern.maxproc
The defaults for the settings we care about on a fresh Snow Leopard Mac OS X 10.6.5 installation are as follows. Each of these are described in the Postgres doc, with some of that info pasted here.



SHARED MEMORY
  • kern.sysv.shmmax
    The most important shared memory parameter. The maximum size, in bytes, of a shared memory segment. Desirable settings are in the hundreds of megabytes to a few gigabytes. Must be a multiple of 4096 or else your setting will be ignored.
    Some possible values:
    ‣ 67,108,864 = 67 megs
    ‣ 134,217,728 = 134 megs
    ‣ 536,870,912= Half a gig
    ‣ 805,306,368 = Upwards of a gig
    ‣ 1,073,741,824 = 1 gig
    ‣ 1,610,612,736 = 1 1/2 gigs
    ‣ 2,147,483,648 = 2 gigs
    ‣ 4,294,967,296 = 4 gigs
    ‣ 6,442,450,944 = 6 gigs
    Reasonable value: Depends on your database. From several megs to several gigs.
    Snow Leopard default: 4194304
  • kern.sysv.shmmin:
    Minimum size of shared memory segment (bytes).
    Reasonable value: 1
    Snow Leopard default: 1
  • kern.sysv.shmseg
    Maximum number of shared memory segments per process.
    Reasonable value: Need only 1 segment. But setting may be higher.
    Snow Leopard default: 8
  • kern.sysv.shmmni
    Maximum number of shared memory segments system-wide.
    Reasonable value: Like SHMSEG plus room for other applications
    Snow Leopard default: 32`
  • kern.sysv.shmall
    Limit on the total amount of shared memory in the system. Measured in 4 kB pages.
    Reasonable value: ceil(shmmax/PAGE_SIZE).
    Snow Leopard default: 1024
SEMAPHORES
  • kern.sysv.semmni: 87381
    Maximum number of semaphore identifiers (i.e., sets).
    Reasonable value: At least ceil((max_connections + autovacuum_max_workers) / 16)
    Snow Leopard default: xxx
  • kern.sysv.semmns:
    Snow Leopard default: 87381
  • kern.sysv.semmnu:
    Snow Leopard default: 87381
  • kern.sysv.semmsl: 87381
    Snow Leopard default: 87381
  • kern.sysv.semume: 10
    Snow Leopard default: 10
PROCESSES
  • kern.maxprocperuid: 266 //
  • kern.maxproc: 532 //
When setting these by way of a sysctl.conf file, you must include all 5 of the following settings, or else your settings will be ignored.


  • kern.sysv.shmmax
  • kern.sysv.shmmin
  • kern.sysv.shmmni
  • kern.sysv.shmseg
  • kern.sysv.shmall
The Postgres doc suggests the following for Mac OS X. Notice that these suggestions happen to be precisely the default values in Snow Leopard (as seen above), so you need not bother at all. While these defaults let you install and run Postgres, you must raise shmmax and shmall when your databases grow.
  • kern.sysv.shmmax=4194304
  • kern.sysv.shmmin=1
  • kern.sysv.shmmni=32
  • kern.sysv.shmseg=8
  • kern.sysv.shmall=1024
The ReadMe that accompanies the Postgres 9 one-click installer suggests the following for a 2-gig MacBook. These settings seem way too large to me for a computer with only 2 gigs of memory.
  • kern.sysv.shmmax=1610612736
  • kern.sysv.shmall=393216
  • kern.sysv.shmmin=1
  • kern.sysv.shmmni=32
  • kern.sysv.shmseg=8
  • kern.maxprocperuid=512
  • kern.maxproc=2048
And the ReadMe notes:
  • kern.sysv.shmmax must also be a multiple of 4096.
  • (kern.sysv.shmall * 4096) should be greater than or equal to kern.sysv.shmmax. 
Bruce Momjian, one of the core contributors, has a wealth of information and advice on such matters, especially his Administration class slides.


The upshot seems to be that you should check for these things:
  •  kern.sysv.shmmax >= several million [several megabytes]
  • (kern.sysv.shmall * 4096) >= kern.sysv.shmmax
  • (kern.sysv.shmmax / 4096) = 0   [even multiple of 4096]
If you do decide to edit your syscntl.conf file, be sure to make a backup copy first. You'll find that file in the hidden folder /etc. Finding and editing such hidden files is easily done using the excellent Path Finder tool, a Swiss-army knife for Mac programmers and geeks.

My Mac's settings seemed to meet their requirements, so I made no changes. Perhaps earlier versions of Mac OS X required such changes. This was the only speed-bump I encountered, and it turned out to be a non-issue in my case.