The author

Dullroar
a heaping plate of rambled eggs

My personal web site. You're probably not interested.

Outdoor Adventures

Trip reports from over the years.

A Bit of Weather
Colorado Colorado Trail Weather

We got rained and hailed on all day, but luckily this one came through after we set up camp.

From my through-hike of the Colorado Trail in the summer of 2016.

Walk Through Blooming Tundra
Colorado Colorado Trail

A great example of the beauty of blooming tundra.

From my through-hike of the Colorado Trail in the summer of 2016.

A Ptarmigan and Chicks
Colorado Colorado Trail

I got to see ptarmigan and their chicks multiple times. It was always a joy.

From my through-hike of the Colorado Trail in the summer of 2016.

Walk Through Wildflowers
Colorado Colorado Trail

A walk through a beetle-killed forest with beautiful wildflowers blooming.

From my through-hike of the Colorado Trail in the summer of 2016.

Big Country
Colorado Colorado Trail

The Colorado Trail isn't all walking through high mountains and summit views. Here is a view in cattle country in the Cochetopa Hills.

From my through-hike of the Colorado Trail in the summer of 2016.

Early Morning Elk Herd
Colorado Colorado Trail

A true privilege to get to see this. I watched them for half an hour.

From my through-hike of the Colorado Trail in the summer of 2016.

A Tad Windy
Colorado Colorado Trail Weather

A bit windy (it got worse!) in the Collegiate Peaks Wilderness.

From my through-hike of the Colorado Trail in the summer of 2016.

Lake Ann Pass
Colorado Colorado Trail

View from the summit of Lake Ann Pass.

From my through-hike of the Colorado Trail in the summer of 2016.

"Typical" Campsite
Colorado Colorado Trail

Here is a "typical" campsite, this one at Lake Ann.

From my through-hike of the Colorado Trail in the summer of 2016.

Hope Pass
Colorado Colorado Trail

View from the summit of Hope Pass on Independence Day.

It was notable because there was no wind and I hadn't seen anybody to this point, even though it was a beautiful three-day weekend!

From my through-hike of the Colorado Trail in the summer of 2016.

Colorado Trail Slides
Colorado Colorado Trail

Just a sampling of random pictures from the trip.

From my through-hike of the Colorado Trail in the summer of 2016.

Changes
Colorado Colorado Trail

Follow along as my beard grows and my weight drops 15 pounds in 28 days.

From my through-hike of the Colorado Trail in the summer of 2016.

My Colorado Trail Stats
Colorado Colorado Trail

Some statistics from the hike.

  • Age: I turned 56 on-trail, same day as reaching the CT high point
  • Height: 6'1"
  • Weight at start: 190
  • Weight at end: 175 (maybe I should write a "Thru-Hike Diet" book!)
  • My home's elevation: ~700'
  • Training: Running 6-9 miles 3x/week, some longer on weekends (up to 20 miles), lots of hill work
  • Base pack weight: 11.3 lbs—pack, shelter, sleeping, cookwear, etc.
  • Total pack weight: 25 lbs—average, this is with the "consumables"
  • Start date: 6/25/2016
  • End date: 7/22/2016
  • Days to complete: 28
  • On-trail days: 26, so two zero days
  • Nights in town: 8 (counting zero days), Breck, Frisco, BV, Salida (2x), LC (2x), Silverton
  • Resupplies: Frisco, Twin Lakes (parcel delivery—too much food!), Salida, LC, Silverton
  • Longest mileage: 26.2, 1st day!
  • Shortest mileage: 7.6
  • Average mileage: 18.4, counts only on-trail days, a bit off because my GPS watch had issues on two days—throw out the outliers and figure just under 20
  • Longest day on-trail: 11.8 hours
  • Shortest day on-trail: 3 hours, doesn't count zero days, obviously
  • Average day on-trail: 8.5 hours
  • Average pace: 2.2 MPH, very steady, +/- 0.3 MPH
  • Average daily climb: ~3,160'
  • Most rain and hail: San Juans south of Molas Pass, but Holy Cross Wilderness gets honorable mention
  • Most wind: Collegiate West segments 3 and 4—I literally cursed the wind one day!
  • Best wildlife: Collegiate West
  • Most unexpected beauty: "Cattle country" in the Cochetopa Hills
  • Worst beetle kill: Upper Cochetopa Creek
  • Biggest disappointment: Arriving at the San Luis Peak saddle too late in the afternoon to summit
  • Easiest hitchhike: To/from Cottonwood Pass to BV
  • Hardest hitchhike: Monarch Pass to Salida (ended up calling a shuttle both ways), Silverton to Molas—I could write a whole post complaining about why these were hard, but will not, because it will sound like sniping/whining
  • Long-term friendships made: Two, and have continued to adventure with both since
  • Wonderful people met: Too many to count, really—and I am one cynical SOB, so when I say "really," I mean it!
  • Unneeded anxieties: Too many to count, really—creek crossings, dry sections, bears, hitchhiking, food amounts, gear, etc., etc., etc.

Colorado Trail Weather
Colorado Colorado Trail Weather

I wanted to give something back to the Colorado Trail after through-hiking it in 2016, so I wrote a page to let you find the current weather anywhere along the route.

Mt. Massive
Colorado 14ers

Climbed Mt. Massive (14,421') using the southwest slopes route.

It was steep but I did it in great time, reaching the summit in under three hours, and felt good about it. However, the descent ended up killing my knees. Good thing it was the last peak I had planned for the trip.

Round Trip
8.8 miles
Time
6:00
Starting Elevation
~9,980'
Maximum Elevation
14,421'
Total Elevation Gain
~4,300'

Click on any picture below to see the slides full-sized.

Mt. Harvard
Colorado 14ers

Climbed Mt. Harvard (14,420') from the standard route along North Cottonwood Creek and Horn Fork Basin in the Collegiate Peaks Wilderness Area northwest of Buena Vista, Colorado.

There was still quite a bit of snow near the summit which made for a few interesting detours. Got to see a goat on the summit, probably from the same herd as the one I saw on nearby Mt. Columbia the year before.

I remember being saddened by a young man who turned back within a few hundred yards of the summit because of the snow and (he thought) the impending weather. I passed his camp on the way out, and he looked bummed. But, each person has to do and learn these things by experience, and this was one of his. Maybe next time he'll take the chance and power on through. On the other hand, I didn't get a lot of summit shots, because the weather was turning.

Round Trip
13.6 miles
Time
7:30
Starting Elevation
~9,730'
Maximum Elevation
14,420'
Total Elevation Gain
~4,830'

Click on any picture below to see the slides full-sized.

Iowa Peak
Colorado 13ers

I climbed Iowa Peak (13,831') from a high camp at 12,000' in the Missouri Basin of the Pine Creek Valley in the Collegiate Peaks Wilderness Area northwest of Buena Vista, Colorado.

I started out wanting to climb Missouri Mountain (14,067'), but had to turn back due to post holing through wet, soft snow at around 13,200' below the ridge between Missouri and Iowa.

I then descended to a shelf and traversed south and climbed Iowa Peak from the southeast. While it wasn't my original goal, it was a beautiful day, lots of flowers were blooming in the tundra, and I got a lot of good pictures. I also got to scope out the Apostles to the west, which are on my list.

Round Trip
6 miles
Time
5:00
Starting Elevation
12,000'
Maximum Elevation
13,831'
Total Elevation Gain
2,550'

Click on any picture below to see the slides full-sized.

Mt. Belford and Mt. Oxford
Colorado 14ers

Climbed Mt. Belford (14,203') for my second time along with Mt. Oxford (14,160').

I started from a high camp at 12,000' in the Missouri Basin of the Pine Creek Valley in the Collegiate Peaks Wilderness Area northwest of Buena Vista, Colorado. Ascended to Mt. Belford by Elkhead Pass, and then crossed over to Mt. Oxford via the connecting ridge, and returned the same way.

Round-Trip
9 miles
Time
6:27
Starting Elevation
~12,000'
Maximum Elevation
14,203'
Total Elevation Gain
~3,530'

Pine Creek and Missouri Basin
Colorado

Backpacked up the Pine Creek valley to Missouri Basin in the Collegiate Peaks Wilderness Area northwest of Buena Vista, Colorado.

Camped at 12,000' for three nights and hiked out on June 30. During my stay I climbed Mt. Belford and Mt. Oxford and also Iowa Peak.

Distance
11 miles
Time
5:38
Starting Elevation
~9,635'
Maximum Elevation
~12,000'
Total Elevation Gain
~3,400'

Click on any picture below to see the slides full-sized.

Mexican Mountain
Utah

John, Sally and I made a trip to the San Rafael Swell in Utah in the spring of 1999.

It was a great trip, and I had a blast. Among the things we did was climb Mexican Mountain (part of a wilderness study area). While it is only 6,393' high, it was still a workout! We car camped overnight, and then drove to where the rugged "road" ended near the mountain. From there we walked around to the north side of the mountain and crossed the San Rafael River (about thigh deep) at Mexican Bend (4,500') to begin the actual climb.

We then climbed the north slopes up a talus cone until we reached a very short stretch of actual rock climbing (5.0 at most!). Of course John scaled it without even blinking, then they coached me up over it and on we went, wrapping around to the east side of the mountain between the main bulk and a cool tower to the east. From there it was up through some very smooth, wonderful sandstone (but then, in Utah, what isn't?) I remember being on one slope that was about 40° and just rolled away for a long way down, and thinking "If I slipped, I wouldn't stop", but the sandstone has such excellent traction that slipping would be out of the question.

We kept kept circling around the summit while continuing our climb, scrambling higher and around to the south side. We finally broke out onto a platform between a minor summit on the south side and the main summit cone on the north. The views were amazing out over the desert. Quite different than the trees and snow of Colorado, but beautiful and breathtaking nonetheless. We rested for a while and then began the descent, with John doing his usual rocket-propelled descent, leaving Sally and me, mere mortals, to wend our way down on our own. While descending the talus cone, I got quite sick—once again, dehydration and heat getting to me (note: take more than two liters for desert excursions—even "day hikes"). After a rest (waving to John off in the distance), we continued down, finally crossing the river and getting back to their SUV (and more water!) As soon as I rehydrated I was fine and ready for more. A good day.

Round-Trip
~4 miles
Time
~5:00
Starting Elevation
~4,500'
Maximum Elevation
6,393'

Click on any picture below to see the slides full-sized.

Goblin Valley
Utah

Part of my trip with John and Sally to the San Rafael Swell in Utah included a jaunt to Goblin Valley State Park.

It is full of "goblins" or "hoodoos," rock formations caused by the erosion of the soft sandstone in the area. Not as majestic as Bryce or Zion National Parks to the southwest, certainly not a "climbing" story, it is still a fun place worth showing, with the rock formations being very accessible. We spent a few hours scrambling around here, exploring lots of nooks and crannies. It is basically a giant playground.

There were good views of the Henry Mountains to the southwest. I've wanted to go back and climb those—they are so very remote and isolated! I was even corresponding with a professor at one point who does yearly scientific studies in that range, but decided I didn't want to spend my vacation catching and tagging bats at night. Yikes!

Click on any picture below to see the slides full-sized.

Arches
Utah

A canyon, an arch, and a frog.

After climbing Mexican Mountain John, Sally and I drove south of I-70 into the southern part of the San Rafael Swell, scrambling in the rock formations in Goblin Valley State Park, seeing some ancient Indian petroglyphs and basically just checking out some of Utah's remote regions. Then we went down to Moab, and camped by the Colorado River at the Big Bend campground. This is located right across the road from Lighthouse Tower, an impressive spire John had climbed in the past.

The following morning John and Sally went rock climbing on Wall Street, dropping me off for a day hike across the Colorado River from Arches National Park up the Negro Bill Canyon. My first goal was Morning Glory Arch, about 2.5 miles up the canyon from the trailhead (4,000'). It was a quick, easy hike there, and I took a few pictures and then decided to get away from the pedestrians and hike further up the canyon, hoping to get off trail (of course!)

The hike was pleasant, but hot. On the way up the canyon I passed by a little slot canyon choked with willows. I marked the place in my mind and went on, eventually reaching some large rock formations and just scrambling around on the rocks for a few hours about five miles up the canyon. Then I came back down and checked out the slot canyon. There was a "trail" of sorts, but nothing that got any real traffic. Which was surprising because it was quite pleasant, with a little stream trickling through it. And anywhere in Utah with water and some protection from the all day sun means plants and trees, which meant there was some shade, too.

When I reached the end of the canyon, there was a little water pool being fed by a trickle falling from a ledge above it. The sound of the tinkling water and the coolness of the shaded pool compared to the desert around me made it the perfect resting spot. Especially since no one else was there. Well, almost no one. There was a big frog in the pool, with just his (or her!) two eyes above the water, checking me out. We watched each other and the surrounding area for about 45 minutes, and I am telling you, I envied that frog's life, and felt special just getting to share its space for that time. Sound silly? It wasn't—this is truly one of my favorite memories of anything I've done outdoors, anywhere. No goals, no big hike to get somewhere, just hanging out in a beautiful place with my new friend, the frog. It was peaceful, quiet and pretty—and if you can't take the time to notice that when you're outdoors, why are you there?

I finally decided to leave the pool and the frog in peace, and went back the way I came, merging onto the main trail and heading down, where I met John and Sally coming up looking for me. We then went and lunched in Moab, and started the drive back home to Colorado.

Round Trip
~7 miles
Time
~5:00
Starting Elevation
~4,000'
Maximum Elevation
~4,800'

Click on any picture below to see the slides full-sized.

Mt. Elbert
Colorado 14ers

Mike and I climbed Mt. Elbert, the highest peak in Colorado, on my 38th birthday.

It was the best birthday, ever! We had returned back to the Arkansas River valley after having to abort our backpacking trip through the Collegiate Peaks due to the shoulder strap on Mike's pack breaking. After camping overnight at Twin Peaks campground off Independence Pass road, we got started up the Black Cloud trail around 8:00AM on the morning of the 15th, a beautiful day. We were taking the non-standard route up the south side of the mountain in the hopes to do something different than the standard "pedestrian" route up the east side, and to avoid as many people as possible.

The trail started at 9,690' and went up a beatiful creek drainage, passing by an old mining cabin at one point. It was a steep trail. According to Dawson's Guide to Colorado's Fourteeners, there is 4,743' of elevation gain in the four miles from trailhead to summit, yielding an average grade of 22%. As Dawson calls it, "a grunt" (and rates even the summer climb as advanced, and supposed to take 10 hours round trip). But we were both feeling very fit, and took it fast, breaking above timberline in less than an hour. From there the trail went over a series of false summits (each a build up of hope followed by disappointment at seeing yet another, higher summit beyond). We finally summited at 11:00AM, which was a very good pace!

The views from the summit were beautiful. The Arkansas River valley was below us to the east, with Mt. Sherman (14,036') in the Mosquito Range beyond. To the north, Mt. Massive (14,421') rose impressively. It really is massive, the summit comprising the largest single area above 14,000' in the conterminous United States. To the west and south of us was the Sawatch Range, including my beloved Collegiates stretching away to the south. Of those, La Plata Peak (14,336') was the most striking, with numerous snow-filled couloirs on its north face capturing our imagination. We both want to go and do a late spring snow climb up one of those! The other thing I remember is that the little flowers in the tundra were in full bloom, and the air was so thick with the pollen from these tiny little blossoms that it smelled like honey. Incredible.

The only thing marring the summit was the crowd. 14ers are typically crowded with people, everyone checking off yet another peak bagged, and as the highest mountain in Colorado (and second highest in the conterminous U.S.) Mt. Elbert gets a lot of people. This time it was made worse by a church youth group, which had dragged a large wooden cross penitential-style up the mountain, set it up at the summit, and then gathered around it singing hymns. I have no problem with worship or hymn singing in the appropriate setting, but on top of a mountain most people are looking for solitude and quiet. If they can't appreciate the direct evidence of God's handiwork in silent reverence, why even bother making the trip? An even worse crime was they then ate, throwing candy wrappers and other trash away on the summit. So much for respecting creation! Grrr...

We then hiked down the way we came, reaching the trailhead by 1:30PM. So much for Dawson's 10 hour round trip estimate! Mike and I were both clicking that day, and it had just felt great to be out and about, fit, and doing what we were doing. Besides on the summit, we had not met more than six people on the trail we took (all of them on the way down), so that part of the plan was successful, too. We then decided to change our itinerary, and instead of camping that night we drove to Buena Vista and had a soak in a sauna and dinner at a Mexican restaurant, planning our backpacking trip for the next day. A great birthday!.

Round-Trip
11 miles
Starting Elevation
9,700'
Maximum Elevation
14,433'

Click on any picture below to see the slides full-sized.

Mt. Yale
Colorado 14ers

Mt. Yale was the first 14er Mike and I summited together.

We drove up to Buena Vista on a Friday night in the summer of 1997, and then on up Cottonwood Pass road until we reached the Denny Creek (not to be confused with Denny Gulch) trailhead at 9,900'. We then parked the car and backpacked up the trail a mile or so, camping in a nice open area near 10,800'.

On Saturday morning we were up and off for the summit. It wasn't that far, perhaps two to two and a half miles, but with a 3,400' elevation gain in that distance, it was a workout. Mike was going fairly slow, because he had just fought off his second bout with cancer, and after the chemo only weighed something ridiculous like 135 pounds (on a 5'11" frame!). Once we had summited he told me the thing that kept him going was a pretty girl on the trail just ahead of him - he followed her best "asset" all the way up! Ha!

Since this was summer in Colorado, it started to rain while we were on the summit, and being too stupid to get off (we were too inexperienced to be lightning leery yet), we just sat up there for a bit with some other stupid people and got wet. Then it was a soggy hike back down to camp. We were very tired, and Mike slept for a while that afternoon. We then cooked dinner and were crawling exhausted into our sleeping bags for the evening when Mike discovered that thanks to his discount tent, his sleeping bag had gotten wet in the latest rainstorm. So the decision was made to bail out, and we packed up camp and hiked down the trail in the rain with only one flashlight between us, going very fast and getting back to the car in about 45 minutes (without a twisted ankle!). That was fun!

Round-Trip
9.5 miles
Starting Elevation
9,900'
Maximum Elevation
14,196'

Click on any picture below to see the slides full-sized.

Mt. Bierstadt
Colorado 14ers

You have to love a mountain whose name, translated from the German, is "Mount Beer City."

I climbed this mountain in October, 1996. It was my third 14er (Grays and Torreys being my first two). I didn't write a trip report because it wasn't worth one and I wasn't writing them then. I followed the basic pedestrian route, starting at Guanella Pass, hiking through the swampy beaver meadows between the pass and the base of Bierstadt, almost getting a boot sucked off in the mud there. From there it was a straightforward slog up 2,300' of tundra and talus slope to the top. Typical 14er crowd scene on top, of course, but the weather and views were terrific.

In the photo of me on the summit, you can tell it was early days for me—I'm wearing jeans and still in my 80s-style moussed hair.

Round-Trip
7 miles
Starting Elevation
11,669'
Maximum Elevation
14,060'

Click on any picture below to see the slides full-sized.

Fun and Sundry

Mostly things I've written to entertain myself.

A Bridge to Nowhere
Humor

A little story about management, goals and quality assurance.

I have recently finished reading a new set of kōans for programmers. I have long loved reading kōans in general. I like their paradoxical stories, even if I don't necessarily achieve enlightenment while reading them.

Given the nature of my personality, today I thought of the idea of writing "anti-kōans." Just like what Despair.com has done with (de-)motivational posters, I wondered, "How could you come up with kōans that bring home a lesson, a real, valuable lesson, but about something that is not necessarily positive or enlightening?" Below is my first example, something I call a "grōan" (pun intended). I am thinking of writing a series of them called "The Hateless Hate" (based on The Gateless Gate—get it?)

A bridge to nowhere

Once the prince came to Jiǎogēn, the humble monk tasked with making sure all foot, rider and wagon traffic moved on the kingdom's roads as quickly and directly as possible. The prince declared his vision for a new bridge, which would become the main bridge by which the many merchants, pilgrims and others in the kingdom would cross the dangerous, raging river to reach the imperial city, replacing the sturdy, if old and rather ugly stone bridge that was used now. The dream of the new bridge that the prince described to the monk was indeed impressive. It was to be built entirely of spider silk, renowned for its strength and versatility, and would be the envy of all rival kingdoms near and far.

"Surely it will take many years to construct such a worthy edifice, especially considering the need to use materials unfamiliar to any of the kingdom's craftspeople, requiring them to learn new skills," thought the monk. Caution would obviously be warranted, since if the bridge failed it would send whoever was on it—merchants, visiting dignitaries, perhaps the king himself—plummeting to their deaths in the rapids below. Therefore, Jiǎogēn asked how much time would be available to create such an impressive structure.

"Three months," answered the prince.

"Three months?" replied the monk, unsure that he had heard correctly. "Surely you mean three seasons? Or perhaps even three years?"

"Three months. It must be ready in time for the opening day of the large summer fair in the city. It will be the showcase of our power to all around."

"But in just three months time, we might just possibly be able to gather enough spider silk to build a simple rope bridge," protested the monk.

"The treasury has been instructed to allow you to purchase whatever you need. You will have your silk."

"And none of our engineers are familiar with working with spider silk," continued Jiǎogēn.

"Here are some scrolls describing how monks in other kingdoms have been able to build simple ladders using such silk," replied the prince. "A ladder is just like a bridge—a bridge between different heights instead of different sides! Such examples must be similar enough to be useful to you. And I have great confidence in your ability to take these descriptions of such puny attempts and expand them far beyond simple toys and build a bridge impressive enough to make our kingdom famous far and wide!"

"But even if we can purchase all the silk we need, and even if I can draft every worker in the countryside, and even if I can decipher these arcane scrolls and extract any useful lessons from them, three months gives us barely enough time to build such an edifice. There won't be time to make sure it can withstand the massive amount of people, animals, carts, chariots and wagons that will be crossing it to enter the city for the opening of the festival!", cried the monk. "How can we possibly open the new bridge if it hasn't been given a fair trial? Who will test it?"

"The travelers are aware of their role," calmly answered the prince, as he strode from the room.

Upon hearing this, Jiǎogēn became benighted.

The Case Against Meat
Food Health Morals

Why I do not eat meat or fish.

Note: This is not a polemic, nor is it a request for comments. I am not interested in your reasons for eating meat. I am not interested in your arguments about why I should eat meat. It is likely that if you are reading this it is because you asked for my reasons, and this is my answer. But that wasn't an invitation to "a conversation" in which you then try to talk me out of it.

Reason 1—My Health

I have always struggled with my cholesterol, and my numbers at my physical in 2016 weren't good. My doctor was talking statins if they don't improve, so I would rather try to make them improve without medication. After being just a vegetarian (not vegan) again for about a month, I had my numbers checked and they were much better. Total cholesterol is still high, but the "ratio" of HDL/Total was great (3.6, with 3.5 or below considered "excellent").

Reason 2—The Environment

I won't go into depth here, but just list the usual suspects. CAFOs are obviously the top one, and it's a big one for me. The water usage and runoff from livestock (or "protein," grrr) raisers worldwide is huge as well, especially as water is going to get more scarce. Just the general amount of fuel and resources needed and pollution produced in general by animal husbandry are also on my mind.

The depletion of fishing stocks in the oceans has led me to believe there is no environmentally sustainable way to continue to consume fish at the levels we do, which is unfortunate because I will miss sushi (although veggie sushi is pretty damned good).

And that doesn't even count things like antibiotics in feed, feeding fish and other animals to livestock, etc.

Reason 3—Spiritual Honesty

Sometime in the past five years, sitting there contentedly communing with my dog Merv, I've known I am sitting there with another consciousness, a "being." Others may argue about whether animals have souls or not, but I know from being with my dog that they feel happiness, excitement, pain, sorrow and other emotions. And if a dog like mine, with his small brain, can feel those things, then probably a cow can, too. Or even a chicken.

At some point, it is easier to not try to draw the line of "This brain is small enough to eat—it's OK" versus "This brain is too sophisticated, we can't eat it." In a similar manner, I once had a boss whose five-year old daughter sounded out "Dolphin safe" on a tuna can, and asked what that meant. When her mom explained that the tuna was caught in a way that kept dolphins from being killed, she asked, "But Mom, what about the tuna?," and became a vegetarian on the spot.

Combine that with my own realization that I would never eat meat again if I had to butcher the animals to do it. I don't even like cutting up a fish any more. There is just too much of a feeling of, "This was a living, breathing being with God's life implanted in them," and then when I look up and see my dog I wonder, "How can I judge the difference?"

Some would say, "You could just 'go organic,'" but in the end the animal is still dead. It was raised specifically to be killed and consumed.

"Our beef comes from happy cows"—Whole Foods sign in Boulder
"They aren't happy any more."—me

We're not talking, "But if you were starving, you could." Because I am not starving, and likely never will be. So it isn't a matter of survival—I don't need to eat meat to survive.

Finally, some may argue animals are placed under our dominion to be "sacrificed" to us, but I'm just not that important, thanks.

Reason 4—Simpler Life

I want to continue to simplify and downsize, and not eating meat will help with that. Less food in the fridge. Less types of cookware needed to produce a dinner. A lot less money! I loves me a good ribeye, but don't feel like taking out a second mortgage to have one these days. But if I am then "settling" on "just" sirloin, why not settle on saving the cow's life entirely, and have a grilled portobello instead?

Also, I honestly enjoy a lot of meatless meals. So if I don't need to eat meat in any given meal, then why eat it—just because of convention? "Sorry, cow—you had to die because I have to have some kind of meat on the plate to satisfy society, and your species was in the rotation for tonight." That doesn't cut it with me. Basically, we eat meat because of tradition. But hey, we used to own other people because of tradition, too, so tradition isn't enough of an argument.

Reason 5—Other Humans

After I first wrote this, my friend Aaron pointed out that another reason is that the meat processing industry is not worker-friendly and there are lots of injuries, repetitive stress disorders and the like, so while those workers are there volunarily, I still don't feel comfortable supporting something that hasn't changed much since The Jungle was written. Especially not when you hear about how the meat is "processed" in those places, anyway.

Summary

I am certainly not doing this to make life easier. In this part of the country eating out as a vegetarian is boring—pasta prima vera, salads (but not Caesar, Cobb or anything like that), bean burritos, a few uninspired Chinese dishes...and you're done. Or drive to the Indian restaurant in Columbia, 30 miles away.

I will probably still eat meat from time to time. Mmm, Mom's home cooking. But I am going to try and be much more mindful of it. I am going to be aware it is not just "meat" or "protein" (nice euphemism), but a dead animal, an ex-being—and consume it with that thought. Being aware that there are health, environmental, spiritual and lifestyle aspects to eating another being are all part of it.

But if it is cooked by someone else and served to me—thinking specifically of eating at my parents, for example (and not how to "game the system" by simply having someone else do it for me), then I will eat it. Similar to how some (not all) Buddhist monks will eat whatever is put into their begging bowl, even if it is meat, because to do otherwise would be to waste the food, which is worse, since then the animal was killed for effectively no reason.

But I don't want to personally increase the demand for meat.

Notes

See this:

A Plea for the Animals: The Moral, Philosophical, and Evolutionary Imperative to Treat All Beings with Compassion

Also see this.

Christian Buzzword Bingo
Humor Religion

Something I wrote while studying to be a pastor (yeah, really).

Concerts I've Seen
Music

Mostly for my own reference, here is a list of concerts I've been to over the years.

The following is a list of all the concerts that I've seen so far.

  • Elvin Bishop—in 1977 opening for E.L.O. (below). I wasn't a fan, especially not of Fooled Around and Fell in Love, although I remember I Strut My Stuff from the concert favorably, and have since heard and liked him on Prairie Home Companion.
  • E.L.O.—on January 19, 1977 in Denver's McNichols arena. What can I say? I was into "progressive/art rock" in the 1970s. I got better.
  • Ramsey Lewis—ca. 1977-8 in Ft. Collins with my folks. I grew up hearing a lot of music in the house; my parents actually have a pretty wide spectrum of tastes in music. Ramsey Lewis was on the "hi-fi console" "playlist" a lot. We got to see him live as a family and it was good.
  • John Sebastian—opened for Fleetwood Mac (below). Don't honestly remember him, which is too bad.
  • Firefall—opened for Fleetwood Mac. They're from Boulder. I grew up in Boulder and came of age at their peak, but at the time I thought they sucked. I still think so, actually Bring out Fleetwood Mac!
  • Bob Seger—I was reminded by a thread on Facebook that the Fleetwood Mac concert also had Bob Seger as an opener. You can determine my feelings about Bob Seger then and now by the fact that I had forgotten that I'd seen him live.
  • Fleetwood Mac—at CU's Folsom Field, May 1, 1977, during Rumours tour. They were great. It started to rain and given the exodus from general admission on the field, I actually ended right up at the stage while they continued to play (they were under cover - we were not).
  • Leon Redbone—got to see the great crooner of songs of times past at the University of Missouri's Jesse Hall ca. 1982 or so.
  • Dave Brubeck—my earliest musical memory ever (five? younger?) is the still-awesome Time Out album, which my parents listened to, a lot. Got to see Brubeck with them at MU's Jesse Hall ca. 1982-3.
  • Canadian Brass—saw them with my folks at M.U.'s Jesse Hall sometime in the early-to-mid 1980s.
  • 3—got dragged to this undead reincarnation of E.L.P. around 1988 in KC by my then sister-in-law. It was dreadful.
  • John McCutcheon—saw him sing and play the hammer dulcimer at the KC Unitarian Church in the late 1980's/early 1990's.
  • The Millions (x2)—this band you've never heard of was out of Nebraska and was popular in the club scene in KC at the turn of the '90s. Saw them open for Pylon (below) and then saw them in an illegal show (they'd call it a rave now) in an abandoned warehouse in the industrial bottoms of KC—until the cops came and closed the thing down.
  • Pylon—saw them in a club in KC around 1990. R.E.M. had this to say about them for their cover of Pylon's "Crazy" on Dead Letter Office —"I remember hearing their version on the radio the day that Chronic Town came out and being suddenly depressed by how much better it was than our record."
  • Neil Young—one of my long-time favorite artists. Saw him at Memorial Hall in Kansas City, KS, on January 12, 1989, with my Dad (another forever fan) and friend Gabe. He did an acoustic set and then came on with "the Restless" (Frank Sampedro, Ben Keith, Rick Rosas, Chad Cromwell) and rocked the house.
  • Robyn Hitchcock and the Egyptians—opened for R.E.M. (below). This was the So You Think You're In Love era.
  • R.E.M.—at Kemper Arena in KC on March 4, 1989 during the Green tour. Went with my friend Gabe and my dad.
  • Feelies—absolutely one of my favorite bands. I was introduced to them by my friend Gabe in anticipation of the Lou Reed concert (below). I am a fan for life.
  • Lou Reed—at Memorial Hall, Kansas City, KS, April 10, 1991 for the New York tour. Great show. My ears rang for days.
  • Kathy Mattea—opened for Clint Black (below) Much better than Mr. Black.
  • Clint Black—got dragged to this show at the Missouri State Fair in Sedalia in 1990. I do not like mainstream country (roots country—Hank Williams, Johnny Cash, yes. Alternatwang, yes. Commercial country, no.) It sucked. What really pissed me off was for an encore he and his band did a great blues number and I remembered thinking, "You could have played like that all along?
  • Joey Skidmore—my friend Calvin's stepbrother, Joey taught me to drink Stingers. Creator of the song Buttsteak. Saw him in a club in KC for the Welcome to Humansville album release party.
  • Emmy Lou Harris—saw her at a free outdoor concert at Crown Center sometime in 1990-91.
  • Chris Isaak (x2)—what a great performer! Puts on a great show, has a great voice. Saw him first in Boulder in the early 1990s at the Fox on the Hill in Boulder. Then at Sybase's 10th anniversary Christmas party in 1995.
  • Clarence "Gatemouth" Brown—opened for Taj Mahal and Michelle Shocked (below) at the University of Colorado's Macky Auditorium sometime in the early 1990s. First time I'd ever heard of him (sorry), but he put on a good gig. Too bad I was distracting waiting for the next two acts.
  • Taj Mahal—I've liked Taj since I was a teenager, so getting to see him and Michelle Shocked in one evening (and even play together during an encore) was excellent.
  • Michelle Shocked—I've loved Michelle ever since I heard the Texas Campfire Tapes. Considering her opening acts and that Alison Brown was in her band at the time and the fact this was at the height of her albums (the tour was in support of Arkansas Traveler, if I recall correctly), all I can say is, great show! Too bad you weren't there.
  • Joe Sample—I had never heard of this jazz pianist until my friend Guido dragged my friend Gregg and me to his show at Yoshi's in Oakland in 1995. Excellent venue, excellent sushi, excellent seats, excellent friends, excellent music.
  • Brian Ferry—what a suave m—f—. Great voice, too. Saw him at Macky in Boulder on November 13, 1999. They did Avalon, that was enough.
  • Elton John—saw him with Billy Joel (below) on April 12, 2001 at Kansas City's Kemper Arena. Good show. They did a set with both bands on stage, then each did a solo set (and each covered one of the other's songs), and then they did a finale set with both bands on stage again. Fun. While I am not an ongoing fan, his greatest hits (first volume) was one of the first LPs I ever bought with my own money. The stuff up until he split from Bernie Taupin still stands on its own.
  • Billy Joel (x2)—my lovely wife has been a Billy Joel fan forever, and I have taken her to his concerts twice (and she has since seen him a third time). First at Kemper Arena in KC on April 12, 2001 for the Face to Face tour with Elton John (above). Then on December 6, 2007 we saw him at the Sprint Center in KC from the 11th row. I have to admit that while I am not the world's greatest Billy Joel fan, I do enjoy his concerts.
  • Alabama 3 (x2)—saw them in Norwich, England at a club called the Waterfront around 2003, then again at London's Astoria on December 14, 2004. All I can say about them is "Wow". Along with the Feelies, probably my favorite band in this list.
  • Anne McCue (x2)—Les and I love (love, love, love) Anne McCue. She's an awesome songwriter and a great guitarist. I got to see her at The Living Room in NYC on August 9, 2010, and she even signed a CD for Les, who was very jealous for not getting to go. Then Les saw her in Champaign, Illinois a few years back, and we both went to see her at the Columbia Public Library on August 24, 2017.
  • Sarah Jaffe—Sarah played at the Living Room before Anne McCue (she didn't "open" for Anne—the club doesn't work that way). I had never heard (of) her before, but am a fan after listening to her set. If you like singer-songwriters (and I do), then you'll like Sarah.
  • Erin Bode, Tony DeSare and Bucky Pizzarelli, and Andra Suchy, all at the Fabulous Fox in St. Louis for our first Prairie Home Companion show.
  • Interpol—opening act for U2 (below). Good from what we heard, although their "sound" got a bit repetitious by the end of the set. Maybe it was just me.
  • U2finally got to see them on July 17, 2011, during their 360° tour at Busch Stadium in St. Louis at Busch Stadium. Awesome show, even if it was freakin' HOT (90+ degrees in the shade, even after 9:00pm).
  • The Bluegrass Martins (x4-5)—they actually live around the corner and throw a free block party every year. Now that's being a good neighbor! Great music.
  • Liverpool Legends—a Beatles tribute band we saw at the Main Street Music Hall in Osage Beach on March 8, 2013. We took the three youngest as part of Gloria's birthday present (she's a big Beatles fan). They were much better than I expected, and it was a good time had by all.
  • Moon Hooch—opening act for They Might Be Giants (below). Think "house" music but with two saxes and drums. I went in not even caring who was opening for TMBG and walked out a huge fan of these guys.
  • They Might Be Giants—took the three youngest on March 14, 2013, to see them at the Blue Note in Columbia. A great show full of "nerd rock" songs and lyrics. The crowd was very familiar with their music and there was a lot of singing and dancing along. Good times.
  • Lyle Lovett—on May 13, 2014, at Jesse Auditorium at MU. Took my folks (long-time fans) to see him play with an acoustic band. Great show with great seats (front row balcony!)
  • Joe Newberry, Pokey LaFarge and Steve Wariner—on Prairie Home Companion at the Fabulous Fox Theatre in St. Louis on June 14, 2014.
  • Dave Rawlings Machine with Gillian Welch (x3)—first at the Sheldon in St. Louis on June 25, 2014. Quite possibly the best sounding concert I've ever attended. John Paul Jones from Led Zeppelin was their mandolin player! Then again on April 17, 2016 at the Blue Note in Columbia. Finally as "Gillian Welch and Dave Rawlings" at the Folly Theater in Kansas City on September 14, 2018.
  • Sheryl Crow—another bucket list for both Les and me. Saw her at the Missouri Theatre in Columbia on October 9, 2015.
  • Arlo Guthrie—saw him with Les and my folks in Columbia at the Missouri Theatre on February 24, 2016. Lots of fun.
  • Margaret Glaspy—good singer-songwriter, we saw her open for the Lumineers (below).
  • Andrew Bird—I've long loved his quirky music since the Squirrel Nut Zippers and Bowl of Fire in the 1990s. He opened for the Lumineers and played a few songs with them as well. Great musician!
  • Lumineers—Les and I went to see them at the Sprint Center in KC on January 24, 2017. Quite possibly the best concert I've ever been to.
  • B-52s—Erin and I went to see them at the Uptown Theater in Kansas City on October 22, 2017. Fred was a little cranky, but it was a fun show. My only real beef was it was too short (about 90 minutes exactly).
  • "Weird Al" Yankovic—Jon and I went to see him at Jesse Auditorium on the MU campus on June 8, 2018. This was him doing mostly original (but still highly derivative) songs and it was amazing! I can honestly say I was blown away and didn't expect the level of musicianship he and his band displayed, or how good of a singer he is! And of course it was a lot of fun.
  • Joan Jett—Les and I saw her open for Heart at the Starlight Theatre in Kansas City on October 8, 2019. She was fantastic, a great performer and crowd-pleaser.
  • Heart—This was a bucket list item for me, and I paid for the privilege so that we were not only 4th row center, right in front of Ann Wilson's mic, but also we did the VIP "meet and greet" package. Beautiful show, one of my all-time favs.

Felix the Flying Frog
Humor

A parable about schedules, cycle times and shaping new behaviors.

Editor's note—I would like to give credit for this but do not know the original author, since it was forwarded around the Internet years ago without attribution. If anyone knows who is the original author and can prove it, please let me know to give them their just credit for writing one of the true classics of all time.

Once upon a time, there lived a man named Clarence who had a pet frog named Felix. Clarence lived a modestly comfortable existence on what he earned working at Informix, but he always dreamed of being rich.

"Felix!" he exclaimed one day, "We're going to be rich! I'm going to teach you how to fly!"

Felix, of course, was terrified at the prospect: "I can't fly, you idiot...I'm a frog, not a canary!"

Clarence, disappointed at the initial reaction, told Felix: "That negative attitude of yours could be a real problem. I'm sending you to class."

So Felix went to a three day class and learned about problem solving, time management and effective communication...but nothing about flying.

On the first day of "flying lessons", Clarence could barely control his excitement (and Felix could barely control his bladder). Clarence explained that their apartment building had 15 floors, and each day Felix would jump out of a window starting with the first floor, eventually getting to the top floor.

After each jump, Felix would analyze how well he flew, isolate on the most effective flying techniques and implement the improved process for the next flight. By the time they reached the top floor, Felix would surely be able to fly.

Felix pleaded for his life, but it fell on deaf ears. "He just doesn't understand how important this is...." thought Clarence, "but I won't let nay-sayers get in my way."

So, with that, Clarence opened the window and threw Felix out (who landed with a thud).

Next day (poised for his second flying lesson) Felix again begged not to be thrown out of the window. With that, Clarence opened his pocket guide to Managing More Effectively and showed Felix the part about how one must always expect resistance when implementing new programs.

And with that, he threw Felix out the window. (THUD)

On the third day (at the third floor) Felix tried a different ploy: stalling, he asked for a delay in the "project" until better weather would make flying conditions more favorable.

But Clarence was ready for him: he produced a timeline and pointed to the third milestone and asked, "You don't want to slip the schedule, do you?"

From his training, Felix knew that not jumping today would mean that he would have to jump TWICE tomorrow...so he just said: "OK. Let's go." And out the window he went.

Now this is not to say that Felix wasn't trying his best. On the fifth day he flapped his feet madly in a vain attempt to fly. On the sixth day he tied a small red cape around his neck and tried to think "Superman" thoughts.

But try as he might, he couldn't fly.

By the seventh day, Felix (accepting his fate) no longer begged for mercy...he simply looked at Clarence and said: "You know you're killing me, don't you?" Clarence pointed out that Felix's performance so far had been less than exemplary, failing to meet any of the milestone goals he had set for him.

With that, Felix said quietly: "Shut up and open the window," and he leaped out, taking careful aim on the large jagged rock by the corner of the building. And Felix went to that great lily pad in the sky.

Clarence was extremely upset, as his project had failed to meet a single goal that he had set out to accomplish. Felix had not only failed to fly, he didn't even learn how to steer his flight as he fell like a sack of cement...nor did he improve his productivity when Clarence had told him to "Fall smarter, not harder."

The only thing left for Clarence to do was to analyze the process and try to determine where it had gone wrong. After much thought, Clarence smiled and said: "Next time...I'm getting a smarter frog!"

Here's a poster I created for this parable:

Floaters: The Story of an Eye
Health

A little something I put together about my experience with a retinal tear.

Internet Rules
Internet

This list is meant to be a set of talking points for engaging with family and friends and reminding them of the implications of interacting on the internet with companies and each other via services such as Facebook, Twitter, and Gmail.

The point isn't to stifle all interaction, but instead to try to raise awareness of the consequences with an eye towards increasing "privacy prudence." It is easy to toss out "If you are not the customer, then you are the product," but what does that actually mean? It means:

  • Any information you give a company will be kept forever.
  • This includes via offline means like warranty cards, sweepstakes, subscriptions and the like.
  • Any information you give a company will be used to sell you things.
  • Any information you give a company will be used to deny your claims.
  • Any information you give a company will be sold to other companies.
  • You won't know all or any of the companies that bought your data.
  • Any information you give a company will end up in (the/a/many) government's hands.
  • Elected officials, political parties, religious organizations and charities that you give information to will act the same as any company with your data.
  • Any information you give a company will end up in criminal hands.
  • Credit monitoring, deleting accounts and class action lawsuits can't undo the damage done.
  • Just because a company says they deleted your data at your request doesn't mean they did.
  • Any information you give a company can and will be used against you in a court of law.
  • You can't know all the pieces of information you give to a company, because some will be silently collected by invisible means.
  • Every web page you visit is giving a company information about you.
  • Every email you receive is giving a company information about you, or trying to, via embedded images that tell whether you opened the email and unique web links (personalized URLs, or "PURLs") that track how effective the email was in getting you to engage with the company.
  • Any information you give a company will be aggregated with information you've given other companies and the government, yielding ever more insights into your life, health, mental state, family, relationships, business and leisure, as well as those for your family and friends.
  • Those other companies may change their relationship to you based on the data they bought—your insurance payment may go up because of your credit card purchase patterns or payment history.
  • Your search results and news headlines will be limited by algorithms that "know what you want."
  • Any information you give a company probably sells some family and friends down the river.
  • Your family and friends have sold you down the river.
  • No one sold their family and friends down the river on purpose—you were just exchanging emails (data), social posts (data), calendar invites (data) and links to interesting things (data) with your contacts (data).
  • "Anonymized data" usually isn't.
  • It is getting harder and harder to surf the internet anonymously, no matter how many browser plugins you install—look up "browser fingerprinting."
  • Any terms of service for using a site, software or service can be changed at whim, invalidating all past promises by a company.
  • Silently append "for now" to any statement made by any company in regards to privacy or data collection.
  • Even companies founded on the premise of preserving privacy "forever" should be heard as always saying, "for now."
  • Mergers and acquisitions trump terms of service, every time.
  • There is no such thing as a "good" company, there is only maximizing shareholder value.
  • Selling and using your data against you maximizes shareholder value.
  • Privacy can be given away, but it can't be bought back.
  • There is no way to control any of the above, other than to not play.
  • Not playing is data, too.

Contributions in the same spirit are welcome. Thanks to Aaron and Mark for their feedback.

Scenes from My Past Lives
Humor

Some scenes from my past life as a software consultant.

While they did not all happen in one place, and while some of it is a little exaggerated, I swear most of it is true! Or true enough.

Systemantics
Humor

Get your daily dose of systems theory humor.

Tech Stuff

Various bits of tech I've figured out and want to remember.

Ten Steps for Linux Survival
Linux

I wrote a book! If you'd like to check it out, click on one of the following:

Always Read Generated Code
C#

This is one of my "A-ha" moments.

Consider how a lot of interfaces work—load an object from some remote system, make some property changes on the object, then save it back to the remote system. The pseudocode looks something like:

if (needToChangePropX == true)
{
    LogChange("PropX", obj.PropX, newPropXValue);
    obj.PropX = newPropXValue;
    objChanged = true; // To keep track if anything changed
}

if (needToChangePropY == true)
{
    LogChange("PropY", obj.PropY, newPropYValue);
    obj.PropY = newPropYValue;
    objChanged = true; // To keep track if anything changed
}

// And so on...

if (objChanged == true)
{
    obj.Save();
}

Besides just the general ugliness that has to come from checking all the values to determine if something applies or not, there are those three repetitive statements in the body of each if:

  1. Log something (because without logging debugging this thing is almost impossible, especially production problems).
  2. Change the actual thing you want to change.
  3. Mark that you have changed the list item.

That last piece is because you don't want to save the list item on every property change. The round-trips and version history would kill you.

Even if you extract out everything to a method, you're having to manually call that method for every property change.

The above works, but it sure is ugly (as interfacing with legacy systems often is).

But for interfaces that are based on service references or .edmx files in Entity Framework, there is a better way.

While researching something else, I actually was looking at the generated code in Reference.cs that gets created when doing a service reference...The first thing that caught my eye but wasn't immediately useful is that all the classes in the generated code are partials—hmmm. More on that at the end. But the second thing that caught my eye was this:

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Data.Services.Design", "1.0.0")]
public string PropX
{
    get
    {
        return this._PropX;
    }
    set
    {
        this.OnPropXChanging(value);
        this._PropX = value;
        this.OnPropXChanged();
        this.OnPropertyChanged("PropX");
    }
}
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Data.Services.Design", "1.0.0")]
private string _PropX;
partial void OnPropXChanging(string value);
partial void OnPropXChanged();

Well look at that. Every property in that generated context class raises OnChange events! At first, I thought I could use the partials and just override those for each property. But that is a lot of work. But see that this.OnPropertyChanged("PropX") at the end of the setter? THAT is interesting! Let's go look at that:

[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Data.Services.Design", "1.0.0")]
public event global::System.ComponentModel.PropertyChangedEventHandler PropertyChanged;
[System.CodeDom.Compiler.GeneratedCodeAttribute("System.Data.Services.Design", "1.0.0")]
protected virtual void OnPropertyChanged(string property)
{
    if ((this.PropertyChanged != null))
    {
        this.PropertyChanged(this,
                             new global::System.ComponentModel.PropertyChangedEventArgs(property));
    }
}

It checks to see if the public PropertyChanged property has an event handler in it, and if it does, it calls it, passing in both the list item object itself (the list item's current state after the change) and the property name that changed! Bingo!

So, in the loop where I am reading each item, I now just do this up front:

obj.PropertyChanged +=
(object sender, System.ComponentModel.PropertyChangedEventArgs e) =>
{
    var changedItem = (RemoteObjectType)sender;
    Log.Trace($"{e.PropertyName} changed to " +
               "{changedItem.GetType().GetProperty(e.PropertyName).GetValue(changedItem, null)}");
    objChanged = true;
};

Thanks to closures, the handler gets access to the objChanged flag. And using reflection I can log based on the property name passed in.

So now the code to check and set a property simply looks like this:

if (needToChangePropX == true)
{
    obj.PropX = newPropXValue;
}

But we still get logging and the flag set telling us the list item has changed.

How cool is that?

Code Challenge!
C#

Here is a bit of fun I wrote for a class years ago.

Empty anonymous collections in C#
C#

This week I had a need to create an empty list with groups of three items (name, customer number, external entity id) in one place in a method, and use it further down in the same method.

This is transient data used only in this single method. Since I am only using it in one place, creating a class or struct seemed like overkill. (And yes, it indicated a refactor was necessary, and I later did so, but stay with me—la la la can't hear you).

My first thought was to use a tuple:

var x = new List<Tuple<string, long, int>>();
...
x.Add(new Tuple<string, long, int>("Jones, David", 12345, 1));

Of course the problem with that is while it is strongly typed, the names when I consume it later are Item1, Item2 and Item3. Not to meaningful.

OK, then I can use dynamic, and give it meaningful names when I add an item to the list

var y = new List<dynamic>();
...
y.Add(new { name = "Jones, David", custno = 12345, entityid = 1 });

The problem with that is then I don't get strong typing, so if I mistype one of the member names when I am accessing it later, I won't know until run-time. Not good.

This then came to me. It works just dandy, and gives me an empty list of the appropriate type with both meaningful member names and strong typing:

var z = (from x in new List<int>() select new { name = "", custno = 0L, entityid = 0 }).ToList();
...
z.Add(new { name = "Jones, David", custno = 12345, entityid = 1 });

Because it is selecting from an empty list, it creates an empty list, but strongly typed exactly as I need it! Cool? Or horrifying?

Strongly-typed empty anonymous collection

Faking Web Service Calls
Linux

Every once in a while it is nice to be able to mock up something like a Web service without having to write code first.

Perhaps you just don't want to spend the time writing the service yet, no matter how trivial. Maybe you don't have the appropriate software installed or configured. Whatever. If your local Web server or your ISP is running on a *IX/BSD OS, then you can use native "UNIX" file systems to mock up Web service "GET" methods, including parameterized URLs.

Note: The following technique will not work for FAT/NTFS file systems due to the restrictions on special characters like "?" in file names.

Let's say your client code is going to want to build URLs of the type:

http://yoursite.com/objectname?querystring

For example:

http://yoursite.com/customers?Dick%20Tracy

You can simply create test files in any format you want (XML, JSON, etc.) and push them to the root of your host and name them accordingly. For example, you could create a file named customers?Dick%20Tracy, e.g.,

$ touch 'customers?Dick%20Tracy'
$ ls
customers?Dick%20Tracy

The single quotes are required.

Now you should be able to do something like the following (Objective-C code, map the example to your own programming language to make it work for you).

NSURL *url = [NSURL URLWithString:@"http://localhost:8000/customers%3fDick%20Tracy"];
NSDictionary *customer = [NSDictionary dictionaryWithContentsOfURL:url];

Works great! And because there are so few special character restrictions in file names on *IX file systems, you can name your files to represent multiple query string parameters. The above file just as easily could have been named:

customers?lastname=Tracy&firstname=Dick&middlename=

...allowing HTTP "GET" access via:

http://localhost:8000/customers?lastname=Tracy&firstname=Dick&middlename=

It is then easy enough to serve such "requests" using something like:

python -m SimpleHTTPServer 8000

Yeah, this is a hack, but it's saved me time by allowing me to quickly get something I can access using the HTTP protocol's GET method that ultimately mimics the URLs I will be building and sending to a real Web service later. A few minutes of file editing and I have a "Web service" mockup ready for testing from the client prototype, at least for GETs.

Cool.

Finding All Users in A/D Groups
A/D SQL

I am a big believer in using the ADSI query provider for being able to conduct LDAP queries in SQL Server against Active Directory.

You set up an ADSI linked server in your SQL Server (I usually call mine, well, "ADSI") and then use Transact-SQL's OPENQUERY to pass off a query to A/D. You can filter the results in the LDAP query, but you can then also use all sorts of SQL chewy goodness against the result set, such as grouping, ordering, more filtering, etc. All good.

Today I had to solve a specific problem—finding all users in an A/D group, when that group may contain groups, and those groups may contain groups, and so on. What I want at the end is a list of all end users that are in the "root" group, "exploding" all of the child groups so that I get nothing but users at the end, not intervening child groups.

Consider the following example:

  • Apps
    • CRMApp
      • John Smith
      • Mary Jones
    • AccountingApp
      • John Smith
      • Chuck Miller

If I query against the group "Apps" I want to have the following returned:

  • John Smith
  • Mary Jones
  • Chuck Miller

I do not want either the CRMApp nor AccountingApp group names in the results.

The rest of this post will describe the LDAP syntax to use as well as a few tips I often use to filter out "real people" in LDAP query results from service accounts, elevated ids, contacts, and the like. Here is the sample query:

SELECT
    *
FROM OPENQUERY(ADSI,
    '<LDAP://DC=FOO,DC=COM>;
    (&(objectCategory=Person)(objectClass=user)(sn=*)(givenName=*)(title=*)(ipPhone=*)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(memberof:1.2.840.113556.1.4.1941:=cn=Apps,ou=Groups,dc=FOO,dc=COM));
    cn;subtree')

The OPENQUERY query syntax for LDAP via ADSI is as follows:

  • Base distinguished name—the starting point for the query in the directory, e.g., <LDAP://DC=FOO,DC=COM>.
  • Search filters—the LDAP equivalent to the SQL WHERE clause. A unique syntax that takes some getting used to, but there are lots of resources on the internet to learn it.
  • Attributes—the comma-delimited attributes (values) to return with the query results.
  • Scope—the scope of the search, which can be base (only search at the starting point), onelevel (go one level down in the directory hierarchy) or as in our case subtree (recurse through the entire directory from the starting point).

Note: Each of the above query parts are separated from the others by semicolons. You can place the independent parts on different lines as shown above, but you cannot split a given part into multiple lines. In other words, the search filter must be all on one line, as shown.

Let's pull the query apart piece by piece:

  1. The query starts at <LDAP://DC=FOO,DC=COM> and looks recursively down from there (because of the subtree at the end of the query). For performance or security purposes you could change the starting point of the query if you knew it was rooted somewhere deeper in your directory structure.
  2. It looks for for directory entries that:
    • Are a personobjectCategory=Person
    • Are a user (not a contact)—objectClass=user
    • Have a last name (surname, or sn)—sn=*
    • Have a first name (given name or givenName) - givenName=*
    • Have a job title (requires you to fill this in Active Directory, obviously)—title=*
    • Have a desk phone (in our case filled in by our VOIP software)—ipPhone=*
    • Are NOT disabled accounts (note this is basically a "magic number") – !(userAccountControl:1.2.840.113556.1.4.803:=2)
    • Belong to the group (I figure out the group's common name or "cn" first, if you prepend that with the memberof:1.2.840.113556.1.4.1941:= magic string that is what does the recursive lookup through all its member groups and users to find all its members) - memberof:1.2.840.113556.1.4.1941:=cn=Apps,ou=Groups,dc=FOO,dc=COM

At this time the sample query only returns common names (cn), but could return user ids and anything else for that matter just by adding the comma-delimited attributes to the list along with cn at the end of the query. Some common attributes I often include in my results are:

  • cn—common name, for users often equivalent to display name
  • department
  • displayName
  • givenName—first name
  • l—city ("locality")
  • mail—email address
  • manager—if you've filled out the "reports to" structure in A/D
  • postalCode—ZIP code or postal code
  • sAMAccountName—userid or account
  • sn—last name ("surname")
  • st—state
  • streetAddress
  • telephoneNumber
  • title

A few final notes:

  • ADSI needs A/D credentials—when setting up the linked server from SQL Server it has to have A/D credentials to connect to Active Directory, but they don't need to be "special"
  • ADSI can handle custom attributes—if you have added schema extensions to A/D, they can be queried with ADSI.
  • ADSI can't handle repeating attributes—some A/D attributes can hold multiple entries, but ADSI can't query or return these.

That's it. ADSI queries in general are a very common technique I use for various purposes, and this example was a real-world example that came up today. I hope you found it helpful.

For more of my thoughts on LDAP, see this post.

Fixing Dropped RDP Gateway Connections
Windows

These are the two things I had to do to get a Windows remote desktop (RDP) connection to consistently work from a Windows 8 or higher client to a Windows 2012 server via a Microsoft Remote Desktop Gateway (RDG).

Symptoms

  • You set up an RDP connection via a RDG. It works correctly the first time, but all subsequent attempts to login result in the remote desktop client prompting for your credentials but then returning to the RDP connection prompt again without connecting.

And/or:

  • You set up an RDP connection via RDG. It works correctly, but the connection drops and then automatically reconnects every 60 seconds or so.

If either of these are happening to you, read on.

First connection works, subsequent connections don't

The problem comes from how you answer the prompt the first time you successfully connect to the desktop. Depending on your RDG's policy, you may receive a "Logon Message" prompt similar to the following:

Be careful how you answer the two check boxes at the bottom. You must check the first one ("I understand and agree to the terms of this policy") to complete logging in. However, if you check the second one ("Do not ask again unless changes to policy occur") and your client machine is not under the same group policy as the server, i.e., you are connecting to your work's RDG via an RDP connection on your home machine, you will get symptom #1. So leave it unchecked, which means you will have to answer this prompt every time you login via the RDG, but so it goes.

If you have already checked the second check box and are getting symptom #1, the following may be helpful.

Since the following involves editing the registry, you take on all risk by doing this, and for this post I presume you know what you are doing. I am simply outlining what worked for me. Be careful and don't just blindly follow these instructions but use them as a starting point for your own problem determination and correction.

  1. I cleaned up all Remote Desktop entries in the registry dealing with the target RDP server (using its name to search on). These will appear under HKCU\Software\Microsoft\Terminal Server Client\Servers\<servername> (where <servername> is the problematic server). I simply deleted the key for the problem server (only).
  2. There were also entries under HKCU\Software\Microsoft\Terminal Server Gateway\<domain> (where <domain> is the internal A/D domain name of the server I was connecting to). Under that was a key called Messages. This had the message from the above logon prompt cached in it. I simply deleted the key for the specific domain (only).
  3. When I then tried to reconnect I was presented with the above logon message box again. Progress! On a hunch I then checked the first check box but left the second unchecked and from then on was able to reconnect successfully (albeit always having to answer the first check box in the logon message prompt).

This symptom only happened on my home box. On my work laptop it did not, even when the second check box was checked. My theory is that since the work laptop is under the same Active Directory group policy as the RDG server it all "just works," but since the home box is not and has no way of checking policy until it connects, it causes an issue. You would think Microsoft could do better than allowing you to shoot yourself in the foot like that.

Connection keeps disconnecting after 60 seconds

This one seems to be a specific "Windows 8 thing" if you search for it on the 'net (I don't know if it still affects Windows 10). This post has a lot of things to try in the long comment thread. However, the one that worked for me was changing the registry to disable UDP for RDP clients. Note that the original comment that proposes the registry change for this gets the value name wrong. Another post has the correct name in a comment correcting that comment (got that?). To be clear, the value name is fClientDisableUDP ( not tofClientDisableUDP).

Since the following involves editing the registry, you take on all risk by doing this, and for this post I presume you know what you are doing. I am simply outlining what worked for me. Be careful and don't just blindly follow these instructions but use them as a starting point for your own problem determination and correction.

  1. Navigate to HKLM\SOFTWARE\Policies\Microsoft\Windows NT\Terminal Services\Client.
  2. Create a DWORD named fClientDisableUDP and assign it a value of 1.

That should be all there is to it. After making the above change I had a multi-monitor RDP session via the RDG work uninterrupted for hours.

I hope this post helps someone else who is searching for answers to these issues.

Fun with ISNUMERIC
SQL

This bit me, so I thought I would put up a post about it.

See, without looking at the answers below first, whether you can guess which of the following queries return 0 and which return 1 in SQL Server:

SELECT '.',   ISNUMERIC('.')
SELECT '+',   ISNUMERIC('+')
SELECT '-',   ISNUMERIC('-')
SELECT '*',   ISNUMERIC('\*')
SELECT '/',   ISNUMERIC('/')
SELECT 'E',   ISNUMERIC('E')
SELECT '0E0', ISNUMERIC('0E0')
SELECT '$',   ISNUMERIC('\$')

The answers are below (scroll down):















Results:

.    1
+    1
-    1
*    0
/    0
E    0
0E0  1
$    1

The SQL Server help for ISNUMERIC clearly states "ISNUMERIC returns 1 for some characters that are not numbers, such as plus (+), minus (-), and valid currency symbols such as the dollar sign ($)." I thought that means it would consider +1 or 0.1 as numeric. Never in a million years would I have guessed that it meant a plus (+) sign or a period (.) by itself is considered numeric.

You have been warned.

Grokking LDAP
A/D

This article constitutes those little bursts into a higher reality I've had while getting my head around LDAP (five minutes to learn, a lifetime to master!)

Introduction

As I worked on one of two personal LDAP projects I wanted to do, it occurred to me that after reaching a certain level of understanding of LDAP (it was the fourth LDAP project for me), I should share some epiphanies I've had about it over the years.

NOTE: This post is not an LDAP tutorial! Familiarity with LDAP at a conceptual level at least is assumed. Some of this may seem fairly esoteric if you don't understand LDAP at all. This essay is instead a series of notes based on clearing up misconceptions someone who has started to delve into LDAP may have. I struggled under the wrong concepts for quite a while before it started falling into place.

CAUTION: Some of what I say in here isn't "official LDAP speak". In other words, some of the concepts I am talking about are constructs I've made up in my own head that I've found to be useful when dealing with LDAP-based directories, and you won't find them anywhere else (that I've seen).

In addition, I have tried to use certain phrases in very specific ways throughout the article.

  • Object instance—a specific directory entry with a specific fully qualified distinguished name.
  • Object type—the definition of a given set of named attributes defined in a schema. The LDAP schema syntax uses objectclass, but I try to avoid that because of the overloading with object-oriented programming.

Author's note—an early reviewer of this piece objected to my renaming LDAP's objectclass to "object type" and felt the intended audience could make the distinction. He is probably right.

A Hierarchy That's Not a Hierarchy

Everyone approaches LDAP as a hierarchy. This is wrong. That statement may seem like heresy or blasphemy, given the 'obvious' hierarchical nature of an LDAP directory, but I stand by it. The name space of an LDAP directory is hierarchical, but that is all. And note this—the name space hierarchy is not typically used to navigate! Repeat this to yourself 4,096 times until you you get it! The name space hierarchy is not typically used to navigate!

Think about XML name spaces for a similar situation—they are URIs that uniquely name something without necessarily representing a valid URL to which you can navigate. In LDAP, the distingushedName (dn) (pseudo)attribute of an object, a node in the directory, is simply a name, even though it looks like it has more meaning or relationship to the object's contents than it does.

Consider the following distinguished name, in LDIF syntax:

dn: cn=James Lehmer,dc=dullroar,dc=com

If this were the fully qualified distinguished name (FQDN) for an inetOrgPerson object, then commonName (cn) would be an attribute of the object (required, in fact, by the person class, one of the parent class types for inetOrgPerson), and the contents of the cn attribute could be the target of a search. The leftmost part of the FQDN (cn=James Lehmer in our case) is the only component of a FQDN that has to be an attribute in the object with that FQDN.

Note that the domainComponent (dc) attribute which seems to be implied is available from our FQDN is not only not required but it is not even defined as an attribute for the object type inetOrgPerson at all, nor its parents organizationalPerson or person, nor the ultimate parent type top! This is often the case—that some "component" of a FQDN, such as the dc "attributes" in this case (dc=dullroar,dc=com), are not actually attributes of the object being named by the FQDN. Remember, the "N" at the end of FQDN stands for "Name", and it is only that. You cannot imply object content from it.

In some sense, it is better to think of the dn (pseudo)attribute as an opaque string, not having any inherent meaning (as Tim Berners-Lee reminds us we are supposed to think of URIs). Yes, the objects in the directory must be created in a name space hierarchy. But the hierarchy is a name space hierarchy only. You will access the objects via searches that often bypass any component of the FQDN name space entirely.

Remember, you can only search for entries based on their attributes. In the end, dn seems like just another attribute, although in most LDAP implementations you typically can't search on it (you can use it as the starting point of a search, however), which is why I call it a (pseudo)attribute in places. So, to find the above named entry, we would have to use a LDAP search like the following, perhaps setting the starting point for the search at dc=dullroar,dc=com:

(cn=James Lehmer)

NOTE: Some objects do have dc attributes. For those objects, you could perform a search such as the following.

(&(dc=dullroar)(dc=com))

But this search would not work for the inetOrgPerson object, since it nor any of its parent object types include that attribute, and you don't search "down" a hierarchy, you simply search for objects that themselves have specific attributes. What I mean by that is you don't navigate by first finding the node with dc=com and then the node under it with dc=dullroar, etc. Instead, you are looking directly for the node with cn=James Lehmer.

Often, you must alter the schema and add a new object that simply has the attributes that are the parts of the FQDN, so that you can store objects that you can search for using the FQDN, by actually searching on the attributes you stored that comprise the FQDN instead. For example, if we created our own object type myInetOrgPerson, simply added the dc component as an optional component, then filled it in when we entered new entries which would be of both inetOrgPerson and myInetOrgPerson types, then we could do a search as follows:

(&(cn=James Lehmer)(dc=dullroar)(dc=com))

To summarize in one sentence: Do not count on the presence of an attribute in an object just because the FQDN seems to imply that attribute's presence!

Everything You Know Is Wrong

Everything you know about relational, hierarchical and network database management systems is wrong. Wipe it from your head while dealing with LDAP.

A LDAP directory is a database, yes, but it doesn't work like anything you've ever seen before. Navigation, per se, is almost completely missing, other than specifying the starting point in the name space for searches. A directory is relatively slow to update, but almost mind-numbingly fast to search and retrieve from ("search" is the key word in LDAP). Testing in 2001 using Netscape's Directory Server showed random search and retrieve speeds of 8,000+ objects per second from a very middle-range single 800MHz/512MB server hosting a decent sized (600MB) directory over 100Mb switched Ethernet network to multiple multi-threaded clients.

Think Venn Diagrams

Don't think navigation or underlying organization. Think sets and set theory. Think Venn diagrams.

Once items have been added to a directory, barring update or deletion (which, remember, in the directory model is presumed to be much less common than search/retrieval), the primary mode of operation against the directory will be via searches and subsequent retrievals. For these searches, you are basically looking for some set of objects that contain some set of attributes that have (or do not have) the values you are looking for. An example would be helpful. You could have a search syntax as simple as the following.

(description=*)

If run from the base DN (the "root" of the directory name space) this could return a large number of objects of many different types, since lots of object syntaxes contain the description attribute, across object types covering organizations, people and devices. Of course, you would only see those objects that actually had this attribute filled in with a value.

NOTE: The LDAP search syntax defined in RFC 2254 is cryptic, with a sort of reverse reverse Polish notation (RRPN). I think it would be better served by a new language (as with everything, the answer is a new language!) that basically used the English syntax of set notation, such as "union", "intersection", "in"/"not in", etc., rather than the highly symbolic one-off syntax in use by LDAP directory servers now. But I digress.

Some would say searching in a directory is very similar to SELECT in a relational database management system (RDBMS). Those persons would be wrong because in an RDBMS there is an implicit navigation—you (the programmer or user) SELECT against a table or set of tables with a join. You have "navigated" to the data container you want by naming table(s) in the operation, and you have defined what data type you want the content to be by selecting only certain columns from those tables.

In LDAP it's all just in one big directory, and you search starting from a specific starting place in the name space (a starting dn, such as the base DN, or root), choosing to search against the current level only in the name space, the current level and next "down", or the current level and all child levels "downward" in the name space. There are no "tables" or other organizational units. There are simply the name space used to set a starting point for a search and whatever attributes the objects themselves contain.

If you ran the following from the root of the directory name space and said to search the entire directory from there downward, you would get all objects of all types in the entire directory that had anything whatsoever in the description attribute.

(description=*)

This is completely different from the "rectangular" view you'd see from something like the following SQL.

SELECT description FROM my_table

Here you would be presented with presumably uniform description data attached to some specific coherent application "object" (whatever was being described in my_table).

LDAP Schemas Are Easy

LDAP schemas are easy. Once you get the hang of them. In defining LDAP schema elements, you need to understand the following points.

ASN.1 Notation

The schema definition syntax uses ASN.1 notation, and as such, you need to have a unique identifier ("object identifier" or "OID") for everything you define. There is a private enterprise number (PEN) branch you can use to which you can append your organization's enterprise number to define a unique "name space" for your schema additions. You can get an enterprise number surprisingly easily from IANA—if your company doesn't have one, it is good to get if you think you are going to do any SNMP or LDAP development in the future, since both use ASN.1 syntax.

It is then up to you to manage all number assignments "under" your enterprise number. For example, given an enterprise number of 314159, you could divide attributes into category '1', object types into '2', and reserve '3' and above for the future. Then under your first two categories, you would hand out number assignments as you added new attributes or object types to the schema.

The prefix for all private enterprise numbers is:

iso.org.dod.internet.private.enterprise (1.3.6.1.4.1)

So, given our example of an IANA assigned enterprise number of 314159, anything we created in the schema needing an object id would have a prefix of:

1.3.6.1.4.1.314159

Using the prefixing scheme we discussed above, a new attribute could be defined with an object identifier as follows:

1.3.6.1.4.1.314159.1.1

...and an object type called dossierPerson that included that attribute could be defined with an object identifier as follows:

1.3.6.1.4.1.314159.2.1

It is totally up to the organization defining the schema items to manage their own object numbering scheme! To avoid clashing within your own organization, figure some allocation schema and stick with it.

Adding New Attributes

In adding new object types, first you must define new attributes (if you need them). In defining attributes you define an attribute name (and aliases), an identifier (in ASN.1 syntax), a datatype (again, in ASN.1 syntax), whether the attribute is single or repeating, etc. Some examples follow:

attributetype ( 1.3.6.1.4.1.314159.1.1
    NAME 'birthday'
    DESC 'Birth date of person, expressed as a timestamp'
    EQUALITY generalizedTimeMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.24
    SINGLE-VALUE )
attributetype ( 1.3.6.1.4.1.314159.1.4
    NAME 'spouse'
    DESC 'Spouse of the person'
    EQUALITY caseIgnoreMatch
    SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{255} )

Observe that the first attribute, birthday, is a single-value attribute—if a value exists for a given object with this attribute, there can be at most one (which makes sense, given the attribute). On the other hand, the spouse attribute will allow multiple entries (since in some countries polygamy is still allowed, and we want our directory application to be internationalized).

NOTE: It would not be a good idea to use this mechanism for holding ex-spouses, because attribute value order in an object is not guaranteed upon retrieval (so there would be no way to keep the spouse values in order from ex-wife v1.0 through ex-wife v2.0 to current production wife v3.0, for example). For that, it would be better to create new objects for each past or present spouse (containing perhaps marriage start and stop dates as attributes so you can order them), and then have an attribute in your object type that can contain one or more FQDN references to those entries.

Adding New Objects

In adding new object types, after defining any new attributes you may require, you then define the new object types themselves. Object types are basically a collection of attributes that an object is required to contain and attributes that an object may contain. Object types have a type of "inheritance", but it is not an inheritance of behavior or topography (layout in the directory). It is instead simply a union of all the attributes in all the object types that a directory element comprises.

For example, the object type top has only one attribute, a required one, objectClass. All other object types "descend" from top in that all object instances must have at least one objectClass attribute value. In fact, every class instance by definition will have multiple objectClass attribute values, since top only contains objectClass, so an object instance that was just a top object wouldn't be very interesting.

First, let's look at top. Observe the ABSTRACT keyword, denoting an object is never meant to be created and stored as an object of just type top:

objectclass ( 2.5.6.0 NAME 'top' ABSTRACT
    MUST objectClass )

Now, let's look at a "descendant" of top, which is person:

objectclass ( 2.5.6.6 NAME 'person' SUP top STRUCTURAL
    MUST ( sn $ cn )
    MAY ( userPassword $ telephoneNumber $ seeAlso $
        description ) )

If we had an object instance of type person, which is descended from top, we would have the following three required attributes that must exist for our object to even be stored in the directory.

  1. objectClass—from top.
  2. sn—from person, alias surname.
  3. cn—from person, alias commonName.

In addition, we know there would be at least two values for objectClass, containing the values top and person. Then we would also have the ability of optionally storing the following information in our object (defined in the person object type as optional attributes).

  • userPassword
  • telephoneNumber
  • seeAlso
  • description

So when thinking of an object instance, I try not to think of something as being "an object of type person", but instead, as an entry containing data for types top and person. Obviously, it is easy enough to talk of an entry of type person, but that often then obscures which object type a given attribute in an object is defined within. For example, we could say an object of type inetOrgPerson has attribute values for the objectClass attribute, but it would be more accurate to talk about the objectClass attribute coming from the top object type definition of the given object instance.

Why Add New Objects?

You typically define new objects to contain some attributes, pre-existing or that you are also newly defining, that aren't contained in any other object class in the schema. For example, countryName (c) and co (two types of country attributes—the former holds the country display name, the latter the two character country code), are defined as part of the basic schema deployed by default with OpenLDAP. However, they are not defined as being a part of any object type in the same basic schema. So, if you want to store the country someone lives in, you'd have to create a new object class that had c or co or both as (probably) optional attributes. Another reason to add a new object type is to allow values that appear as part of the FQDN of an object to be searched against.

The tasks for schema definition you must do in any LDAP project should include the following:

  • Look at the schemas already available to you, especially those installed by default. On Linux with OpenLDAP, that includes core schemas and schemas to support CORBA, Java, Kerberos, basic tracking of people, and others. Active Directory has similar schemas plus more to support Windows, Exchange and so on.
  • Inventory which attributes you think you are interested in tracking. Most may already exist in the base schema set.
  • Note the object assignments for each—many attributes are used in multiple object types—and try and come up with a coherent minimum set of object types that contain the attributes you need. This set should make some sort of sense, not ending up with an amalgamation of "person" and "device" object types.
  • Catalog which attributes you may need that are not in an object type already. c and co are two good examples of "orphan" attributes with the default OpenLDAP install.
  • Describe which attributes are missing. Give them names, aliases if you wish, and decide which data type (and possibly maximum length) they will hold and the syntax in ASN.1 for it.

Following is an example of a new object type for holding information about people. It is not descended from person or any of the other person-based object types. Instead, it is meant to be used in conjunction with those other types. It is meant to hold attributes not assigned to any other person-based object types. The attributes used are both pre-defined in the base schema (c, co, generationQualifier, houseIdentifier) and custom-defined for this object type (birthday, endDate, child, spouse, startDate, weddingAnniversary).

# Seemingly useful information, especially if you want
# to use the directory as either a contacts or HR
# repository (try to do either without birthday or
# spouse, for example).
objectclass ( 1.3.6.1.4.1.314159.2.1
    NAME 'dossierPerson'
    DESC 'Helpful supplementary contact information'
    SUP top AUXILIARY
    MAY ( c $ co $ generationQualifier $
        houseIdentifier $ birthday $
        endDate $ child $ spouse $
        startDate $ weddingAnniversary ) )

Object Types Are Like Interfaces

Objects in the directory can morph in real-time. When you update objects in the directory you can add new object types to an existing object instance, and that instance can then contain all the required and optional attributes defined in all the object types that the entry contains in the objectClass attribute, including the newly added object type. In this sense, objectClass is simply another object attribute (in fact, part of the top object, as we have seen), and it is a repeating attribute that all objects have at least two values of (one of top and one of some other object type). The set of all the values in the objectClass attribute in a directory entry (object instance) comprises all the attributes the entry (object) can possibly contain. Many if not most of the optional attributes may not contain any value at any point in time, depending on the application.

It is better, however, especially given the generic tools that are out there, to define an object instance in advance as being of all the types you think it may ultimately contain. This allows browsers to see those types defined even if no attributes have been filled in for them yet. It forces the filling in of certain mandatory attributes (if any) for each type when an object instance is created and stored, increasing search options and speed, since mandatory attributes are often indexed.

Be Prepared to Code

You will probably have to write code (unless you're working with an email client).

It is amazingly easy to write generic LDAP browsers (done that, been there). That's why you see so many out there, including one with almost every LDAP toolkit and book. And yet, strangely enough, the one production application almost universally LDAP-enabled, which is email, strongly depends on the client implementation to pick up attributes correctly.

For example, Mozilla publishes an LDAP schema for their Thunderbird client, and they conform to others, because if you point their LDAP address book client at an inetOrgPerson, they will pick up many but not all of the attributes correctly. In fact, some seemingly "standard" attributes such as telephoneNumber may or may not be picked up correctly by a given email client, because they have coded against a specific object class or classes and are looking for specific attributes, and the "obvious" choice wasn't the one chosen.

That said, outside of email there are almost always only two ways to use LDAP:

  1. Through an existing generic browser/editor—For example, the ADSI and LDAP API based tools that come with Microsoft's Active Directory.
  2. Via a custom application you write—With the plethora of LDAP APIs out there including all the Java JNDI and .NET DirectoryServices APIs that abstract a lot of it for that language, it isn't hard to write LDAP applications. I even use the SQL Server ADSI interface a lot, which allows you to issue SQL SELECT statements against Active Directory (with some limitations—it doesn't support multi-value attributes, for example). In fact, if you are adding custom attributes and object types, you are almost assuredly going to be using a custom application if you want to deploy to end users, as opposed to technical personnel only.

A User is Just Another Entry

A "user" is just another directory entry. Some object types can contain (clear text or hashed) password and user id attributes. You can use these to control accessing the directory, both signing in and then controlling access after that. In addition, some object types representing persons, for example, can be made members of other object types representing groups. These can all be used to control access to directory entries in rich ways using access control lists (which are nothing more than a list of FQDNs and the permissions attached to them).

The thing to remember is a "user" is just another directory entry. In fact, in LDAP directories, everything is just another directory entry, including the metadata (just like system tables in an RDMBS are tables that hold information about other database entities, including tables, including themselves).

Conclusion

I hope that was worth the read! I attempted to alter your perception on some specific points about LDAP-based directories. To recap:

  1. Everyone approaches LDAP as a hierarchy. This is wrong.
  2. Everything you know about relational, hierarchical and network database management systems is wrong. Wipe it from your head while dealing with LDAP.
  3. Don't think navigation or underlying organization. Think sets and set theory. Think Venn diagrams.
  4. LDAP schemas are easy. Once you get the hang of them.
  5. Object metadata in the directory can morph in real-time.
  6. You will probably have to write code (unless you're working with an email client).
  7. A "user" is just another directory entry.

Each of these issues cost me some time and some pain before I fully "grokked" it and what it meant when working with LDAP. I hope by reading this, I save you some pain, if not now, then in the future when you land on that LDAP project, or start writing that cool new open source LDAP tool.

LEAD, LAG or Get OVER the Way
SQL

Once I finally learned the OVER clause, something in SQL Server since 2012, it quickly became one of the items in my SQL toolkit.

Running Totals

This is part of this same work, where I started using a technique to turn data from our transactional systems into more of a "time series." But of course, one of the first things people want to have once you get something like a time series are running totals, for things like "velocity" charting (my boss is big on velocity—bad physics jokes elided).

Turns out running totals in SQL are easy now, and the following is lifted straight from StackOverflow. Let's say you have a date, LastTouched, and a numeric column, in this case, Total, and you want to have a running total of Total as LastTouched increases:

SELECT
    LastTouched AS [Last Touched],
    Total,
    -- See Gareth's answer - https://stackoverflow.com/questions/2120544/how-to-get-cumulative-sum#2120639
    SUM(Total) OVER(ORDER BY LastTouched ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS [Running Total]
FROM LemonadesSoldByDate
Running totals
last touched total running total
2018-02-01 5 5
2018-02-02 3 8
2018-02-06 4 12

Diff'ing Row Versions

In the same week, for an unrelated project, I was asked to show differences between rows in a history table. For some applications we have mirror history tables that get the entire row's data inserted on each UPDATE via triggers (I know, how 1990s). But one of the primary uses for this data is to see precisely what changed on each update. It turns out that LAG and LEAD (both also introduced in SQL Server 2012) can be used for precisely this purpose.

To do side-by-side "diffs," with pairs of columns showing current and previous values, you can simply do something like this:

CREATE VIEW LemonadeStandHistoryDiffs
AS
SELECT TOP 10000000 -- To get ORDER BY in a view, because 100 PERCENT never works for me.
    Id,
    ModifiedOnDate AS [Modified On Date],
    Customer,
    SaleAmount AS [Sales Amount],
    LAG(SaleAmount, 1, 0) OVER (ORDER BY Id, ModifiedOnDate) AS [Previous Sale Amount],
    GlassesSold AS [Glasses Sold],
    LAG(GlassesSold, 1, 0) OVER (ORDER BY Id, ModifiedOnDate) AS [Previous Glasses Sold]
FROM LemonadeStandHistory
ORDER BY Id, ModifiedOnDate

Which yields results like:

Side-by-side diffs
id modified on date customer sale amount previous sale amount glasses sold previous glasses sold
1 2018-02-01 Fred 5.00 0.00 5 0
1 2018-02-02 Fred 3.00 5.00 3 5
1 2018-02-06 Fred 4.00 3.00 4 3


Pretty cool. But it makes a really wide result set, with 2x the number of columns as the original, and it is still hard to "eyeball" and see what specifically changed from one row to the next. So instead, we want a "version history." One way is to show a row per changed value, grouped by the modified timestamp:

CREATE VIEW LemonadeStandVersionHistory
AS
SELECT TOP 10000000 * FROM
(
    SELECT
        Id,
        ModifiedOnDate AS [Modified On Date],
        Customer,
        'SaleAmount' AS Field,
        CAST(SaleAmount AS VARCHAR(250)) AS [Current Value],
        CAST(LAG(SaleAmount, 1, 0) OVER (ORDER BY Id, ModifiedOnDate) AS VARCHAR(250)) AS [Previous Value]
    FROM LemonadeStandHistory
    UNION
    SELECT
        Id,
        ModifiedOnDate,
        Customer,
        'GlassesSold' AS Field,
        CAST(GlassesSold AS VARCHAR(250)) AS CurrentValue,
        CAST(LAG(GlassesSold, 1, 0) OVER (ORDER BY Id, ModifiedOnDate) AS VARCHAR(250)) AS PreviousValue
    FROM LemonadeStandHistory
    UNION
    -- and so on...
) A
WHERE CurrentValue <> PreviousValue
ORDER BY Id, ModifiedOnDate

Note casting the numeric (and any DATETIME) columns to VARCHAR. This is to make all the "diff" rows play nicely in the UNIONs regardless of each column's data type.

The above view then produces results similar to this:

Version history
id modified on date customer field current value previous value
1 2018-02-01 Fred SaleAmount 5.00 0
1 2018-02-01 Fred GlassesSold 5 0
1 2018-02-02 Fred SaleAmount 3.00 5.00
1 2018-02-02 Fred GlassesSold 3 5
1 2018-02-06 Fred SaleAmount 4.00 3.00
1 2018-02-06 Fred GlassesSold 4 3


Creating all those UNION statements may be a bit onerous, and I am still thinking of a better way to do the same in SQL without resorting to a PIVOT, although ultimately that may be the way to go.

Combining Running Totals With LAG

Now back to the first example about running totals:

Running totals
last touched total running total
2018-02-01 5 5
2018-02-02 3 8
2018-02-06 4 12


Note there are days where there are gaps, i.e., nothing happened on that day, so there isn't a row for that date in the running total results. For your purposes, that may be OK. But what if you want to have a running total for each day, even if it doesn't change? Using the "exploded dates" table created in the first post, you can do something like the following, combining using OVER for running totals plus LAG. For this example I assume there will never be a gap of more than 30 days:

WITH RunningSalesTotals
AS
(
    SELECT
        LastTouched AS [Last Touched],
        Total,
        SUM(Total) OVER(ORDER BY LastTouched ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS [Running Total]
    FROM LemonadesSoldByDate
)
SELECT
    ED.TheDate,
    COALESCE(
        RST.RunningTotal,
        LAG(RST.RunningTotal, 1) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 2) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 3) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 4) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 5) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 6) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 7) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 8) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 9) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 10) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 11) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 12) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 13) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 14) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 15) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 16) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 17) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 18) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 19) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 20) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 21) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 22) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 23) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 24) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 25) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 26) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 26) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 28) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 29) OVER(ORDER BY ED.TheDate),
        LAG(RST.RunningTotal, 30) OVER(ORDER BY ED.TheDate),
        0) AS RunningTotalClosed -- 0 if there are NULLs at the start of the series
FROM ExplodedDates ED
LEFT OUTER JOIN RunningSalesTotals RST ON
    ED.TheDate = RST.LastTouched
WHERE ED.TheDate >= '2018-02-01'
    AND ED.TheDate <= '2018-02-07'
Running totals with no date gaps
last touched running total
2018-02-01 5
2018-02-02 8
2018-02-03 8
2018-02-04 8
2018-02-05 8
2018-02-06 12
2018-02-07 12

Thanks to Jason Koopmans for his helpful comments on this topic.

LIKE Considered Harmful
SQL

LIKE, am I RIGHT?

We have a vendor database at work to which we've built some custom integrations. It is a document management system. One of the columns in the main table contains object names—folders, files, users, whatever—they store all types of objects in one big table.

If the folder is a "customer-level" folder then it contains the customer's name followed by the customer id in parentheses (I didn't design any of this!) There is no direct access to the customer id by itself, so I wrote some SQL that looks to see if a customer folder already exists using the customer id. It looked like the SQL below:

SELECT
    TOP 1 CASE WHEN Id IS NULL THEN 0 ELSE Id END AS Id,
    Name
FROM Objects WITH(NoLock)
WHERE
    Name LIKE '%(' + CAST(@custno AS NVARCHAR(10)) + ')%'

This has been getting slower and slower as the database grows in size, and was taking 10-15 seconds per run. Not good.

Changing it to use the PATINDEX function instead of LIKE didn't help at all, nor did I expect it to.

Then I changed the wildcard search to instead do an "ends with" search:

SELECT
    TOP 1 CASE WHEN Id IS NULL THEN 0 ELSE Id END AS Id,
    Name
FROM Objects WITH(NoLock)
WHERE
    RIGHT(Name, LEN('(' + CAST(@custno AS NVARCHAR(10)) + ')')) = '(' + CAST(@custno AS NVARCHAR(10)) + ')'

This ran in under a second. I have now rolled it into production.

In retrospect, this result doesn't surprise me. What did surprise me was the order of magnitude difference.

Parsing HTML With SQL
SQL

A friend once sent me a request on how to parse HTML with SQL. Challenge accepted!

Here is the original request:

I was wondering if you knew how to use SQL to parse a column and return multiple individual records where text in that column is started with <b>This is a test</b>. The column might have these tags multiple times.

Sample Column Text...

This is a test
<b>Test</b>
How is it going
<b>Get this Text Also</b>

It should return...

Test
Get this Text also

Here was my initial reply:

You're going to hate me, but you can do this with SQL's XML functionality. I have built a test case that works on your sample set.

NOTE: ALL XML FUNCTION NAMES ARE CASE-SENSITIVE (lowercase), EVEN THOUGH SQL IN GENERAL IS NOT.

Consider the following table, which represents the existing table, which I presumed had the HTML stored as a VARCHAR:

CREATE TABLE dbo.ParseExample
(
    HTMLText VARCHAR(4000) NULL
)

Now we insert the test case into it:

INSERT INTO ParseExample VALUES('This is a test
<B\b>Test</b>
How is it going
<b>Get this Text Also</b>')

SELECT * will bring back what you'd expect.

So, we can convert the HTML into XML (even though it's not valid XML) on the fly using a common table expression:

WITH Converted
AS
(SELECT CAST(HTMLText AS XML) AS X FROM ParseExample)
SELECT * FROM Converted

Now you can do fun things with XPath and XML functions. For example, to just get the <b> elements and their values (the XPath //b will look for bold elements wherever they are in the XML DOM), you can use the query function:

WITH Converted
AS
(SELECT CAST(HTMLText AS XML) AS X FROM ParseExample)
SELECT X.query('//b') FROM Converted

But this isn't useful, because the result still has the <b> tags and also returned everything in one row and column, which isn't what you'd want. So then you use the nodes function to get individual rows for each node that matches the XPath query, and then the value function to extract just the text (and not the tags) from that:

WITH Converted
AS
(SELECT CAST(HTMLText AS XML) AS X FROM ParseExample)
SELECT nref.value('.', 'nvarchar(max)') BoldText
FROM Converted
CROSS APPLY X.nodes('//b') AS R(nref)

The above returns exactly what we are looking for.

Of course, there was one more wrinkle—his actual live data had &nbsp; entities in it, so the XML parser was choking on that with:

XML parsing: line 1, character 9, well formed check: undeclared entity

In follow-up discussions he decided to just do this in the common table expression:

(SELECT CAST(REPLACE(HTMLText, '&nbsp;', ' ') AS XML) AS X FROM ParseExample)

I personally would prefer something more like:

(SELECT CAST(REPLACE(HTMLText, '&nbsp;', '&amp;nbsp;') AS XML) AS X FROM ParseExample)

Anyway, I thought I would document this just for fun.

Stacked Bar Charts in ASP.NET with No Code-Behind
ASP.NET SQL

This is posted as an intellectual exercise someone else may find interesting or useful.

NOTE: This is NOT a suggestion of how to implement something in production (although we do have an app like this there).

Introduction

I am currently working on a project at work that involves among other things gathering metrics for our various scanners across all of our locations. We have a heterogeneous scanner environment (which raises its own issues) with 16 models of scanners from Canon, Fujitsu, Lexmark and Xerox spread across 24 different sites. Some sites only have one scanner, others have two or three. I wanted to show total scanner volume by site, broken down by scanner model at each site. This is a perfect application for a stacked bar chart.

The Problem

I have been using the ASP.NET charting control for other charts in this project, and purely by chance (not design) I had done everything I needed with each chart in pure markup and SQL with no code-behind for the Web page until I needed this stacked bar chart. Then I had a hard time finding any information on the Internet about how to accomplish a stacked bar chart in pure markup (because I wanted to see if I could).

The issue is that for a stacked bar chart to work, there need to be multiple series (in my case, scanner models), and each series has to have the same number of data points, i.e., in my case each series has to have 24 locations. This is because the stacked bar chart is "dumb" and simply stacks the first column in each series on the first bar, the second column in each series on the second bar, and so on. But the locations each only have one to three scanners, so at most each location only has approximately 20% of the scanner models available. Needless to say, a query something like:

SELECT
    Branch,
    ScannerModel,
    COUNT (*) AS Scans
FROM ScanMetrics
GROUP BY Branch, ScannerModel
ORDER BY Branch, ScannerModel

...is going to produce output like this:

This is not what we need, since the output is "ragged" (each location has a different number of scanner models).

Pivot Tables to the Rescue

I thought about it some more and finally figured out what I needed was to have my data in a pivot table. Using the output from that, I could then do the stacked bar chart in pure ASP.NET markup. Here is a view I created to give me the pivot output I needed:

CREATE VIEW BranchScannerScansPivot
AS
WITH Branches AS
(
    SELECT DISTINCT
        Branch
    FROM ScanMetrics
),
Scanners AS
(
    SELECT DISTINCT
        Branch,
        ScannerModel
    FROM ScanMetrics
),
BranchScannerCombos AS
(
    SELECT DISTINCT
        B.Branch,
        S.ScannerModel
    FROM Branches B
    CROSS JOIN Scanners S
),
BranchScannerCounts AS
(
    SELECT
        BSC.Branch,
        BSC.ScannerModel,
        COUNT (SM.ScannerModel) AS Scans
    FROM ScanMetrics SM
    RIGHT OUTER JOIN BranchScannerCombos BSC ON
        SM.Branch= BSC.Branch
        AND SM.ScannerModel= BSC.ScannerModel
    GROUP BY
        BSC.Branch,
        BSC.ScannerModel
)
SELECT
    *
FROM BranchScannerCounts BSC
PIVOT(SUM(Scans) FOR ScannerModel IN
(
    [CANON2200],
    [CANON3080],
    [CANON3100],
    [CANON3300],
    [CANON3320],
    [CANON3570],
    [CANON4035],
    [CANON4080],
    [CANON5035],
    [CANON5051],
    [CANON6010],
    [CANON6050],
    [FUJITSU6010],
    [LEXMARK658],
    [LEXMARK796],
    [XEROX5745]
)) AS Scans

Selecting from this view produces a 24-row result set with exactly the output I need:

It is a "rectangular" grid of values for each combination of location and scanner model.

The Solution

With that data in hand, then producing the stacked bar chart in pure markup is simply an exercise in copying and pasting 16 series (one for each scanner model). Here is the entire Metrics.aspx file:

<%@ Page Title="Metrics" Language="C#" MasterPageFile="~/Site.Master" AutoEventWireup="true"
    CodeBehind="Metrics.aspx.cs" Inherits="ScanMetrics"%>
<%@ Register Assembly="System.Web.DataVisualization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
    Namespace="System.Web.UI.DataVisualization.Charting" TagPrefix="asp"%>
<asp:Content runat="server" ID="FeaturedContent" ContentPlaceHolderID="FeaturedContent">
    <section class="featured">
        <div class="content-wrapper">
            <hgroup class="title">
                <h1><%:Title%>.</h1>
                <h2>Scan statistics.</h2>
            </hgroup>
        </div>
    </section>
</asp:Content>
<asp:Content runat="server" ID="BodyContent" ContentPlaceHolderID="MainContent">
    <asp:Chart ID="Chart1" runat="server" DataSourceID="SqlDataSource1" Height="800" Width="800">
        <Titles>
            <asp:Title Text="Scans by Branch and Scanner Model"></asp:Title>
        </Titles>
        <ChartAreas>
            <asp:ChartArea Name="Branch">
                <Area3DStyle Enable3D="true"/>
                <AxisX Interval="1">
                    <MajorGrid Enabled="false"/>
                </AxisX>
            </asp:ChartArea>
        </ChartAreas>
        <Series>
            <asp:Series Name="CANON2200" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON2200">
            </asp:Series>
            <asp:Series Name="CANON3080" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON3080">
            </asp:Series>
            <asp:Series Name="CANON3100" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON3100">
            </asp:Series>
            <asp:Series Name="CANON3300" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON3300">
            </asp:Series>
            <asp:Series Name="CANON3320" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON3320">
            </asp:Series>
            <asp:Series Name="CANON3570" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON3570">
            </asp:Series>
            <asp:Series Name="CANON4035" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON4035">
            </asp:Series>
            <asp:Series Name="CANON4080" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON4080">
            </asp:Series>
            <asp:Series Name="CANON5035" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON5035">
            </asp:Series>
            <asp:Series Name="CANON5051" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON5051">
            </asp:Series>
            <asp:Series Name="CANON6010" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON6010">
            </asp:Series>
            <asp:Series Name="CANON6050" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="CANON6050">
            </asp:Series>
            <asp:Series Name="FUJITSU6010" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="FUJITSU6010">
            </asp:Series>
            <asp:Series Name="LEXMARK658" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="LEXMARK658">
            </asp:Series>
            <asp:Series Name="LEXMARK796" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="LEXMARK796">
            </asp:Series>
            <asp:Series Name="XEROX5745" ChartType="StackedBar" ChartArea="Branch"
                XValueMember="Branch" YValueMembers="XEROX5745">
            </asp:Series>
        </Series>
        <Legends>
            <asp:Legend Enabled="true" Alignment="Center"></asp:Legend>
        </Legends>
    </asp:Chart>
    <asp:SqlDataSource ID="SqlDataSource1" runat="server"
        ConnectionString="<%$ConnectionStrings:ConnectionString1%>"
        ProviderName="<%$ConnectionStrings:ConnectionString1.ProviderName%>"
        SelectCommand="SELECT Branch, CANON2200, CANON3080, CANON3100, CANON3300, CANON3320, CANON3570, CANON4035, CANON4080, CANON5035, CANON5051, CANON6010, CANON6050, FUJITSU6010, LEXMARK658, LEXMARK796, XEROX5745 FROM BranchScannerScansPivot ORDER BY 1 DESC">
    </asp:SqlDataSource>
</asp:Content>

And here is the output:

Mission accomplished.

Conclusion

Now, I am not saying I think this should be a standard approach. For one, I find it "fragile" as a solution since it requires both the SQL pivot and the chart series markup to know in advance the number and names of the scanner models. Thus, adding a new model would require code changes to both the view and the Web page.

However, I did want to publish this because I have found when working with the Microsoft chart control that often there are fairly straightforward markup-only approaches, but most samples on the Web tend to be a mishmash of markup and code with no good discussion about why a given property is set in markup and another is set in the code-behind. Also, depending on your environment, it may be easier to make changes to markup and SQL views in production than to compiled code. It can certainly be easier to "tweak" markup.

And finally, I just thought it was an intellectual challenge, and hope you found it interesting as well.

Time Series in SQL
SQL

Recently at work I needed to turn different sets of "point-in-time" (PiT) data into time series using SQL Server.

Specifically, given rows in a SQL Server database that represent the current state of an entity, how can we show how long that entity has been or was "in process?" I am sure it is not original, but since I figured this out independently, I thought I would document it here.

Assumptions

The PiT data has to have at least three columns for this technique to work:

  1. Created on—a column indicating the date and time at which the entity was originally created.
  2. Modified on—a column indicating the date and time at which the entity was last altered.
  3. Status—some sort of code indicating whether the entity is still "open" (in-flight, in-process) or else "closed" (finished, expired, cancelled, etc.) This doesn’t have to be a simple yes/no or open/closed status, as we will see.

There are many business entities that have that sort of data—e.g., service tickets in a help desk system, opportunities in a CRM system, loan applications, etc.

Also, we will need a simple table with a series of dates in it. See the next section for that.

Create a Sequence of Dates

I can’t take credit for this—I ripped off the original logic from the accepted answer at this StackOverflow post. However, I just wanted a fixed range, and I chose from January 1, 1980, well before our business data begins, until December 31, 2100, long after I am dead (I will let this be part of someone’s Y2100 problem).

First I created a view:

CREATE VIEW [dbo].[ExplodeDates]
AS
WITH N0
AS
(
    SELECT 1 as n UNION ALL SELECT 1)
        ,N1 AS (SELECT 1 as n FROM N0 t1, N0 t2)
        ,N2 AS (SELECT 1 as n FROM N1 t1, N1 t2)
        ,N3 AS (SELECT 1 as n FROM N2 t1, N2 t2)
        ,N4 AS (SELECT 1 as n FROM N3 t1, N3 t2)
        ,N5 AS (SELECT 1 as n FROM N4 t1, N4 t2)
        ,N6 AS (SELECT 1 as n FROM N5 t1, N5 t2)
        ,nums AS (SELECT ROW_NUMBER() OVER (ORDER BY (SELECT 1)) AS num FROM N6
)
SELECT DATEADD(day,num-1,'1980-01-01') AS thedate
FROM nums
WHERE num <= DATEDIFF(day,'1980-01-01','2100-12-31') + 1

Note that the above generates the given date range of over 44,000 rows fast—under a second (because there is no I/O involved). However, after some testing it seemed like the view would be too slow (not much the SQL optimizer can do with it), so then I created a materialized view and populated it as follows:

CREATE TABLE dbo.ExplodedDates
(
    TheDate DATETIME NOT NULL,
    CONSTRAINT PK_ComplianceChecklist PRIMARY KEY CLUSTERED
    (
        TheDate ASC
    )
)
GO

INSERT INTO ExplodedDates SELECT * FROM ExplodeDates

Now we have a simple, keyed table with all the dates from 1/1/1980 through 12/31/2100. Cool.

Convert PiT Rows to Time Series

For example purposes I’ll use opportunities from Microsoft CRM. However, the concept remains the same regardless of your entity, as long as it has the three columns I mentioned above, i.e., "created on," "modified on," and "status." Let’s create and populate the time series table for opportunities.

CREATE TABLE CurrentOpportunityStatus
(
    StartedOn DATETIME NOT NULL,
    LastTouched DATETIME NOT NULL,
    OpportunityId UNIQUEIDENTIFIER NOT NULL,
    CONSTRAINT PK_CurrentOpportunityStatus PRIMARY KEY CLUSTERED
    (
        StartedOn ASC,
        LastTouched ASC,
        OpportunityId ASC
    )
)
GO

INSERT INTO CurrentOpportunityStatus
SELECT
    FO.createdon AS StartedOn,
    CASE FO.statecode
        WHEN 0 THEN GETDATE()     -- Open
        WHEN 1 THEN FO.modifiedon -- Won
        WHEN 2 THEN FO.modifiedon -- Lost
        ELSE GETDATE()            -- Unknown state code
    END AS LastTouched,
    FO.opportunityid
FROM Your_MSCRM.dbo.FilteredOpportunity FO WITH(NOLOCK)

Note that we use the current date and time if the opportunity is still open, otherwise we used the modifiedon value to indicate when it was closed. For some other entities I have done this for, the CASE statement can get quite long, but at the end, the logic remains:

  1. For "open" status codes, the WHEN returns the current date and time.
  2. For "closed" status codes, the WHEN returns the last modified date and time.

That’s all there is to it.

How Many X Were Open on Any Given Date?

The whole reason I went down this path was then to be able to chart how many of each entity type was open on any given date. That is where the ExplodedDates table comes in to play. For each entity, simply create a view similar to the following:

CREATE VIEW TimeSeriesOfOpportunities
AS
SELECT TOP 10000000 -- TOP used to get ORDER BY in a view
    TheDate,
    COUNT(*) AS OpenOpps
FROM ExplodedDates ED
INNER JOIN CurrentOpportunityStatus O ON
    ED.TheDate >= O.StartedOn
    AND ED.TheDate <= O.LastTouched
GROUP BY TheDate
ORDER BY TheDate

Basically, for each date in the date range, if is is on or after the opportunity’s "created on" date and it is before or on the "last touched" date (either "modified on" or today’s date), then it is counted as "open" on that day. This yields a series like:

TheDate    OpenOpps
1/1/2017    1455
1/2/2017    1455
1/3/2017    1456
1/4/2017    1453
1/5/2017    1463
1/6/2017    1465
1/7/2017    1465
...lots more rows...
1/27/2018    899
1/28/2018    899
1/29/2018    899
1/30/2018    828
1/31/2018    816
2/1/2018     810
2/2/2018     801
2/3/2018     781

Conclusion

Creating the above takes very little time for each business entity. Then, you can start showing graphs that show counts of the various types of entities in your business that are active by date. It is very handy for dashboards and the like.

Trailing Whitespace as "Security"
humor

This is one of those “back in the day” stories, so if you don’t want to listen to this grandpa rock in his chair and spin a yarn, then move along...

My very first job "in computers" was as a computer operator from 1980 to 1985. It was a good job in many ways, especially since it allowed me to go to college during the day and work in the evenings, and I never had to step foot in the school computer lab because I was able to do all my programming for class on my work account.

As time went on a fellow operator and I started creating a bunch of scripts to help us do our jobs, using mostly ISPF screens and first CLIST (the horror! the horror!) and then Rexx as the scripting language to drive the screens.

For reasons that now escape me, we didn't want other operators running our scripts. Yet they were all available, since we were all in the same security group and we didn't have the ability to secure files to our individual ids. What to do? Then I got a "bright" idea.

On MVS at that time script files could be in one of two record formats—80 character fixed length records (to mimic punched cards) and 255 character variable length records. The vast majority of people used 80 character records in their scripts. This was to our advantage. Remember, at this time we were working completely with 3270 "green screen" terminals, most of which were 80 character displays, although there were a few that had 132 character displays. While you could scroll to the right there were no visual indicators that you would need to do so on a long line—you just had to know that there was more data to the right.

So using the psychological expectation by most people that script files are 80 characters wide, the solution was simple. At the top of the file I simply put a comment block, like this:

/************************************************************************/
/* This script written by Jim Lehmer.                                   */
/* It does blah, blah, blah.                                            */
/************************************************************************/

Then, padding those comment lines with spaces I put in an IF statement similar to the following way over to the right of the comment block:

IF USERID() != "jlehmer" THEN
EXIT

Everyone thought we had some sort of special security on the files, because they'd try and run the scripts and the code would immediately exit. No one figured it out, for years. They never thought to scroll to the right, even if they noticed that the record type on the script files was 255 character variable length records.

So now you've heard everything—security through trailing whitespace! The ultimate "security through obscurity."

Training Tesseract
Linux OCR

My notes on training Tesseract for an OCR project at work.

Introduction

Tesseract is an open source OCR tool originally developed by HP and now used by Google and others. The source repository is here:

https://github.com/tesseract-ocr

It is primarily a command-line tool. However, there is a library that is available for programmatic access, which has then been ported to Windows. In addition, there is a .NET wrapper API for it available on GitHub and also in NuGet, so installing it in a Visual Studio project is easy:

https://github.com/charlesw/tesseract

Before Tesseract can be used, it must be "trained." This involves a series of steps that teach it both the font(s) that will be used as well as the language. There are defaults for many languages and standard fonts (Arial, etc.) on the project site itself, including the default one for English:

https://github.com/tesseract-ocr/tesseract/releases

However, if you want to use a different, "non-standard" font, such as OCR-A, you must train Tesseract for that. This document explains how I trained Tesseract for OCR-A for a work project.

Resources

You will want to become familiar with all of the following. Really. Even though in the end the steps I give should "just work," understanding what is going on is helpful.

Training Steps

The following describes what to do to create a new traineddata file, which then goes in the tessdata folder that will be under your project's executable location. In other words, if your project executes out of C:\Program Files\Foo, then the tessdata folder should be at C:\Program Files\Foo\tessdata.

You will have to pick a font (in our example OCR-A) and a three-character language code. The language doesn't have to be real. In our example we use "zzz" as the language code. When you use Tesseract, you tell it which language to load.

All of the following assumes you have access to a Linux box with the following installed:
  • Tesseract
  • Python 2
  • ImageMagick
  • Pango and Cairo (installed with Python)

Step 1—Convert "truth file" to image

This uses the text2img.py script in tess_school. Per the readme:

-text2img.py: Takes a ground-truth text file and automatically generates image files from the text, for use in training tesseract. Everything is hard coded at the moment, no command-line options yet. Eventually I'd like to have this generate the boxfile too.

First I changed the hardcoded language in the script from ka/kat to en/eng, i.e.:

LANG = "en"
TESS_LANG = "eng"

Note: Even though I ended up using "zzz" as the language, it was easier in the interim to work with "eng" because a lot of samples on the Internet assume it.

I then ran it in the shell as follows:

python text2img.py -f text.txt OCRA

Let's pull that apart a bit. Besides the script itself, there are two things of interest:

  • text.txt is the input file. It contains the characters we are going to use for training. In this case I made the file a simple one that had all upper and lowercase letters, numbers, and all special characters accessible on a keyboard. Here are the contents:

    ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz1234567890`[email protected]#$%^&*()-_=+[{]}\|;:'",<.>/?
  • OCRA is the name of the font to output the text as in the resulting image. It has to be the official name of a font installed on the system you are running the script on. On Debian/Ubuntu/Mint flavors, fonts are installed under /usr/share/fonts, and then you have to know what type of font it is, such as OpenType (.otf) or TrueType (.ttf). In our case OCR-A is a TrueType font, and the files are located at /usr/share/fonts/truetype/ocr-a. In that directory are the following files, and by using OCRA we are telling it to use the main OCR-A font:

    • OCRA.ttf
    • OCRABold.ttf
    • OCRACondensed.ttf
    • OCRAItalic.ttf

This brings up another point. When training for fonts, you have to train for normal vs. bold vs. italic vs. bold and italic together all separately. In our case we just want the normal font, so that keeps things simple.

The output of the text2img.py execution will be a file in the same directory named eng.OCRA.exp0.png. That is the language ("eng"), the font we chose ("OCRA"), the "exp0" is used to allow combining multiple training files together (for example, you could have "exp0" be for the normal font face, "exp1" be for bold, etc.).

Step 2—Convert the PNG to a TIFF file

While Tesseract can use PNG files for training, apparently it works better with TIFFs, per the tess_school readme:

-png2tif.sh: Uses ImageMagick to convert the PNG output from text2img.py to TIFF files. Tesseract can read PNG files, but sometimes seems to prefer TIFF.

No changes were required. Just run the script:

./png2tif.sh eng.OCRA.exp0.png

You will then have a file in the same directory called eng.OCRA.exp0.tif.

Step 3—Make the "box" file(s)

This step is where the rubber meets the road. It involves letting Tesseract loose on the image we've produced and see if it can figure out where the characters are and what they are. A "box" file is simply a text file with a simple format:

  • The character Tesseract guessed in a specific location.
  • Four fields that are the coordinates of the character.
  • A page number (zero-relative), which for us is always zero (training multi-page docs is harder).

The make_boxes.sh script generates the box file(s), looking for any .tif files in the directory. The only thing that needs to be changed is the language code, which again I changed to "eng":

LANG=eng

Then you just run the script:

./make_boxes.sh

You will then have a eng.OCRA.exp0.box file in the same directory.

Step 4—Merge adjacent boxes

Tesseract can sometimes "see" multiple characters where there is in reality only one. This script helps fix that, per the readme:

-merge_boxes.py: Merges nearby boxes in a boxfile resulting from tesseract oversegmenting characters. This is a common error that Tesseract makes and this script will quickly fix most instances of this problem.

I had to make no changes to the script. You run it with:

python merge_boxes.py -d eng.OCRA.exp0.box

The -d parameter indicates it's a "dry run" and will just indicate if there were any boxes that needed merging. In my case there weren't. See the script for other parameters in case the dry run indicates there might be boxes needing merging.

Step 5—Align the box file to the truth file

In a typical Tesseract training, you would then go through the box file with a box file editor such as Moshpytt, checking and correcting each and every character. On large box files that is a complete PITA, and from my testing large box files (lots of input characters) didn't seem to significantly increase the accuracy. YMMV. Instead, tess_school has a script that automatically takes in the truth file used to generate the image, and makes sure every corresponding line in the box file is set to the correct character from the truth file. This is handy and very, very time-saving. From the readme:

-align_boxfile.py: Changes a boxfile to match a ground-truth text file. Will abort and complain if the number of boxes doesn't match the number of characters in the file, so run this only after your boxes are in the right places.

I didn't have to change anything in the script. To run it:

python align_boxfile.py text.txt eng.OCRA.exp0.box

It is taking in the truth file we used above, text.txt, and updating in-place the box file we generated, eng.OCRA.exp0.box.

There are other tools in tess_school, but I found its auto_train.sh script to not generate as good of a traineddata file as I got via other means, and I had no use for the other scripts at this time, so we will leave tess_school behind now.

Step 6—Training Tesseract

At this point I wrote a script called trainingtess to finish all the remaining steps in training Tesseract. I won't go through it in detail (the Resources section above has all the gory details). The script is as follows:

#!/bin/bash
tesseract zzz.ocra.exp0.tif zzz.ocra.exp0 nobatch box.train
unicharset_extractor zzz.ocra.exp0.box
echo "ocra 0 0 1 0 0" >font_properties
shapeclustering -F font_properties -U unicharset zzz.ocra.exp0.tr
mftraining -F font_properties -U unicharset -O zzz.unicharset zzz.ocra.exp0.tr
cntraining zzz.ocra.exp0.tr
cp normproto zzz.normproto
cp inttemp zzz.inttemp
cp pffmtable zzz.pffmtable
cp shapetable zzz.shapetable
combine_tessdata zzz.
cp zzz.traineddata /home/youruserid/tessdata/.
sudo cp zzz.traineddata /usr/share/tesseract-ocr/tessdata/.
tesseract zzz.ocra.exp0.tif output -l zzz

You have to make the following changes:

  • Rename the "eng" files that came out of the tess_school work to "zzz", or to change the trainingtess script to use "eng" itself. Your choice. And for your project you may want some other "language" like "foo" or "bar" instead anyway. I also changed the font name in the files from "OCRA" to "ocra" to match Tesseract "standards."
  • Change the font from "ocra" to the appropriate font name for your uses.
  • Change the line that creates the font_properties file appropriately. Its format is:
    • Font name—as used in the file names, here ocra.
    • Italic1 if training for italic font. 0 in our example.
    • Bold1 if training for bold font. 0 in our example.
    • Fixed1 if training on a fixed (monospaced) font. 1 in our example, since OCR-A is a fixed font.
    • Serif1 if the font has serifs. 0 in our example.
    • Fraktur1 if the font is a "Fraktur" font (aka "blackletter" or "Olde English/Gothic" font). 0 in our example.
  • You will also want to change where it copies the output traineddata file.

For input, it will need the .tif and .box files generated by the tess_school scripts to be in the same directory as trainingtess. You then simply run it:

    ./trainingtess

You will get a lot of output files from it, including:

font_properties
inttemp
normproto
output.txt
pffmtable
shapetable
unicharset
zzz.inttemp
zzz.normproto
zzz.ocra.exp0.box
zzz.ocra.exp0.tif
zzz.ocra.exp0.tr
zzz.ocra.exp0.txt
zzz.pffmtable
zzz.shapetable
zzz.traineddata
zzz.unicharset

Out of all those, and out of all this work, the one we're interested in is the zzz.traineddata file. That is what will go in your tessdata directory of your project. The other interesting file is the output.txt file, because that shows the output from the last step in the script, which ran Tesseract with the new traineddata file on the .tif file and had it OCR the image and output what characters it found. If you did everything right, it should be the same characters that are in the image file. If so, you have been successful! Good job, citizen!

Other Issues

I did not have to train Tesseract for "words," which is where the whole language thing really comes into play, with dictionaries and files to help "unambiguate" similar characters, especially when considering kerning issues, e.g., distinguishing "rn" from "m" in a sans serif font. That is more useful when you are trying to OCR entire documents into English, for example.

The Tesseract language files that are on the project site are already pre-trained for some common (mostly sans serif) fonts. If you are trying to do real language processing I would start with those files and hope they "just work." If not, you have a lot of work ahead of you. At that point I would start considering a commercial package.

Using Checksums in SQL to Find Groupings of Like Data
SQL

This is a bit of an edge case and there are probably better ways to approach it, but it took me less than five minutes to whip up and saved someone else in our company a bunch of time.

We have at least two custom apps (neither of which I had any part of writing, BTW—as the post goes on you’ll see why I disavow them) that use denormalized tables that hold both rows for individual items (users in one case, pick list items in the other) and groups (user groups and pick list names, respectively) in the same table. Then, in both apps there is a second “mapping” table that maps the individual items to the group items. Perhaps a sample would be clearer.

Here is the Users table—misnamed, since it holds both user and group definitions:

id name type
1 admins group
2 joe user
3 jim user
4 lusers group
5 jon user
6 jack user
7 jerry user


And here is the GroupUser table—the “mapping” table:

groupid userid
1 2
1 3
4 5
4 6
4 7


With me so far? If you haven’t clawed your eyes out by now, it’s probably because you’ve seen this sort of thing, too. Heinous, heinous stuff. Especially because that Users data table doesn’t just have those three columns in it—no, it has other columns, some of them holding orthogonal data based on what the Type column contains.

Now, consider a really flawed application that we want to rewrite. The application depends on two tables such as the above to hold users and groups in the Users table, with the GroupUser mapping table holding which users are in which groups. For reasons that are too painful to even contemplate the original authors of the app, which deals with reports, decided that each report should have it’s own group for defining who can access it. Worse, they didn’t allow for groups within groups, which means there are literally hundreds of groups (like, almost 800), many of which hold—you guessed it—identical group members, all manually maintained as reports are added or people are hired or leave or change job positions. So obviously as we rewrite the app we want to be able to define such groups once and reuse them across reports.

How to figure out which groups are identical in terms of user membership? Well, one way would be to have the person who administers them sit for a day and figure it all out by hand and hope they don’t miss the fact that some groups may differ from others by only one member (and some of these groups can hold a hundred users in them, so good luck with that). Instead, I came up with the following using T-SQL’s CHECKSUM function.

Here is it in all its glory:

SELECT
    [Group],
    SUM(Checksum) AS Total
FROM
(
    SELECT
        G.NAME AS [Group],
        CAST(CHECKSUM(U.NAME) AS BIGINT) AS Checksum
    FROM GROUPUSER GU
    INNER JOIN USERS G ON
        GU.GROUPID = G.ID
        AND G.TYPE = 'Group'
    INNER JOIN USERS U ON
        GU.USERID = U.ID
        AND U.TYPE = 'User'
    GROUP BY G.NAME, U.NAME
) A
GROUP BY [Group]
ORDER BY 2, [Group]

Basically, this joins each user to their appropriate groups (a row per group/user combo) and takes the checksum of the user’s name (login id, actually). Then those checksums are summed together by group and displayed in order by those sums and then group names underneath that. When looking at the checksums all the rows with the same sum likely hold the same group members.

Now, purists will note there is some chance of collision with checksums, and summing checksums certainly raises the chances of that slightly, but in our case the results that came out look right, in terms of the person who admins all the groups looking at the clusters and saying, “Yes, those groups all have the same members in them.” So I wouldn’t claim this would work across really large data sets but for the purpose at hand it did Just Fine, and five minutes worth of work saved someone a lot of manual cross-checking.

You're Soaking in It
CSS HTML

A long colophon explaining the ideas behind the design of this site.

Introduction

Every few years I redesign this site, usually as an excuse to learn something new. Quite a while back it was built with Pelican, a static site generator. Then I moved to using pandoc, Markdown, Bootstrap and make. Both of those worked OK, but in the end they both required remembering to install a series of toolchains to keep working, and more importantly how those toolchains worked.

Then I decided I wanted to learn more about CSS than what I've picked up on the streets for work. I also wanted to get the "toolchain" down to "just a text editor." Finally, I wanted to eliminate any reliance on Javascript libraries, and just for fun, at least for the main page (this one), to use no Javascript at all. The question then becoming, "How responsive, dynamic and 'modern-feeling' could I make it, sans scripting?" As the title says, "You're soaking in it."

I call the end result SKABS, or "stone knives and bear skins." Written entirely by hand, no scripting except on some auxilliary pages where their functionality requires it (and the use of Javascript on those pages is noted both in the footer and with <noscript> tags), and only one piece of third-party CSS embedded in my styles.css file, but then heavily modified by me.

As time went on, I added some additional constraints. I wanted the main page to emulate a SPA. I wanted it to load fast, so any resources should not be fetched until they are required to be visible. And I wanted user-friendly features, like collapsable menus. Again, all without scripting. Some of my inspiration came from two manifestos I read right around the time I started—Jeff Huang's This Page is Designed to Last, and Daniel Janus's Web of Documents. I don't follow either religiously, but they were in my mind while building this.

More technical details follow—if you are interested I presume you will "view source" (Ctrl+U) and look at styles.css to follow along.

Note: There is also a "style guide" that is really only interesting to me.

Showing and Hiding Without Script

This consists of two techniques. The first is pure HTML—the details and summary elements. These give a way to show some introductory text in a summary text block, and when it is clicked, to expose the rest. This is how the menu works, as well as all the "Read more..." elements following every article's lede. Very, very easy.

The second is a bit more involved, and relies on "the checkbox hack" (search the web for that phrase and you will get plenty of hits). This is how the "tag filters" at the top of the page work. It involves hiding a checkbox but leaving its label exposed. Clicking on the label checks or unchecks the hidden checkbox, and then through CSS other elements can be shown or hidden at will.

Delaying Resource Downloads

As the main page grew bigger and added a lot of images and videos, I noticed every reload was downloading all of them as well, even if they were hidden behind a collapsed <details> tag. Making them display: none didn't stop the downloads, either. Not good, because that would end up bringing down hundreds of megabytes on every page refresh!

For the images there is a well-known hack where, instead of using <img> tags, you can use a <span> or <div> element and either set its content style to the image URL, or set a ::before or ::after pseudo-element's content or background-image to the same. I use both techniques.

For videos it was a bit harder, and ended up being a bit of a hack. The problem is I use Vimeo to host my videos. While I could use the new HTML 5 <video> element and delay downloading by styling it hidden, Vimeo wants to charge me $240 a year for the "embeddable links" required by those. Otherwise, on my free account, I am stuck with embedding <iframe> elements. And <iframe> elements are like <img> tags, in that they download whether they are visible or not. But in this case, there is no easy "set it as the background" hack available as with images. So what to do?

"Every problem in computer science can be solved by adding another level of indirection." In this case, I created a second page and load that in the <iframe>. So of course that second page is loaded as many times as there are <iframe> tags on this page, but at just a bit over 2KB per reference, that's not too onerous. But here's the cool hack—the secondary page embeds an <iframe> itself, but its src attribute is empty! It then uses some Javascript (which is a bit of cheating, but hey, this page isn't using it!) to detect when it is made visible (by a <details> element getting expanded) and sets the src attribute to the appropriate Vimeo URL at that point, and then the Vimeo resources download. I think that's pretty cool!

As of this writing, on page load for index.html, the following network traffic occurs:

  • The page itself, which is by far the "heaviest," as it is over 300KB as I type this.
  • The styles.css file, a tad shy of 20KB at this point.
  • My picture in the upper-left corner—the so-called "hero image" (snork), ca. 28KB.
  • n X the <iframe> placeholder page (at n X 2KB)

In other words, the entire "site" downloads in less than 500KB. And since it is static and everything is cacheable, and it is all text so everything is highly compressable, I feel like that's a pretty good page load goal. Even the skinniest Wikipedia page is around 800KB, and of course most commercial sites really increase the payloads with Javascript libraries, CSS, images, trackers, etc.

Miscellany

A few other comments and thoughts. The first is, to keep bandwidth down and avoid any licensing issues, I use no third-party fonts, not even the so-called "Web-safe" ones. I have restricted myself to serif, san-serif and monospace font-family options exclusively. I also use no special fonts like Glyphicons or Font Awesome, but instead just use Unicode characters for any special glyphs I want, after making sure they render correctly on both desktop and mobile.

The only CSS that isn't hand-carved by me is for the photo slideshows (look at something like Mt. Harvard, for an example). For that I use CSSBox, although I have then heavily modified it to avoid using <img> elements, as well as to add captions.

Getting the "checkbox hack" to layout correctly on mobile was a long and involved process. No matter what I did, on mobile (only) layouts it wanted to reserve space for the checkboxes. Look at the input[type="checkbox"] selector in the stylesheet for everything I did to finally make them hide, hide, hide, dammit!

I only used tables for laying out actual tabular data (like the results from SQL queries), and even so I am not very happy with the results and may keep tweaking them or going to something like <pre> layouts instead (which I did in at least one place already). For one thing, getting tables to hyphenate correctly is a bitch, and because of how the CSS standards are implemented, in English it only works if the words in the table cells are lowercase (really - you then have to text-transform: capitalize those cells to get them back to being, well capitalized). If you don't believe me, try it yourself! But otherwise without hyphenation they look even worse on responsive mobile layouts.

About

I'm Jim Lehmer, a software architect in the middle of nowhere in flyover country. I am also an O'Reilly author.

All content written or created by me and is copyrighted by me unless otherwise noted. Everything expressed here are my personal views, and do not represent my employer.

This site was written using "SKABS" (stone knives and bear skins), that is, plain old HTML 5 and CSS 3. It uses no Javascript, because "The fastest framework is the one you don't download." I consider it both a exercise in self constraint, plus a way to make a site I can edit in Notepad or vi without any additional tooling or prerequisites. Everything has been hand-crafted by me, although I've certainly used inspiration from around the web.

You don't care, but there are style guidelines.