Toto Gamboa

Where Photography & Databases Mix

1.5Terabytes of Photography Gone and Back, and How Windows 7 Installs and Fixes Itself!

leave a comment »

Last Thursday morning, Windows 7 popped an ugly message … cannot read Drive S: and after I closed the message, I immediately opened Windows Explorer and there was no more drive S:. Suddenly, a rush of panic engulfed my senses. It was in that drive that I recently consolidated all my photos, and yeah …. 1.35TB of RAW files since I got into the digital photography madness. I load up event viewer and one item says, “The device, \Device\Harddisk1\DR1, has a bad block.”. Loaded up Disk Management and Drive S just wasn’t there. I rebooted and my PC’s BIOS telling me I had a bad disk.

After a fresh restart, I immediately opened Windows Explorer to check on drive S:. Thankfully it was there but it cannot be opened and accessed. I immediately went on recovery mode. First thing I did was to test if my chances for recovery is high. I ran an old Sandisk tool RescuePro and recovered files without subjecting the faulty hard disk of any write operation. Though very effective, RescuePro just went on and dumped every file it can recover in a single folder with the recovered files named as 00001.cr2, 00002.cr2 …. xxxxx.cr2. After a few files recovered, I realize it would be a nightmare trying to check each file for its content. I cancelled RescuePro and run TestDisk which I had used with my faulty CF and SD cards before. This tool is very advanced in terms of disk/file recovery but its UI is as old as those character based DOS apps of the 80s. Running TestDisk, I was able to peer into the folder structure of the faulty hard drive, and have each recovered to another disk 1.5TB disk. It took me almost 24 hours to have everything recovered. Yeah … ALL files were recovered.

Just as I thought my woes are over, while verifying each folder if it indeed been recovered, Windows 7 PC blue screened. If you are just a street away from me that time, you could have been deaf by the curses you have heard from me. I restarted the PC and it says something about a missing BootMgr. What the !@#$!!!!! Upon further scrutiny, I realized the problem started some couple of years ago when I had this PC freshly formatted when I added some new hard drives. I remember when I had the first HDD upgrade, I set the BIOS to boot on the new 1.5TB HD instead of the switching cables so that boot order would match corresponding disk ports. What happened was I had the following setup:

  • BIOS Boot Order on Device 1
  • Device 0, 320GB, Active Primary D:\, …
  • Device 1, 1.5TB, System, Boot, C:\, ….

Earlier this year, I replaced the old 320GB with another 1.5TB and forgot to reset what was in the BIOS all these times. So I had the following setup:

  • BIOS Boot Order on Device 1
  • Device 0, 1.5TB (new)
  • Device 1, 1.5TB (old)

Without this HDD crash incident, I could not have known that Windows 7 did the following when I had it reformatted right after installing the new 1.5TB HDD some months back.

  • BIOS Boot Order on Device 1
  • Device 0, 1.5TB (System, C:\, C:\Windows)
  • Device 1, 1.5TB (Boot)

As you can see, BIOS tells the PC to boot from Device 1. Since it has crashed, it could NOT find the necessary boot info, thus I got the “BOOTMGR is missing” message. I attempted to BCDEdit, but the app hangs as it accessed the faulty drive. I have tried Windows repair and all to no avail. Windows repair only managed to fix the partition issue but it does not repair BOOT miscue. All these until I got Hanselman’s blog on BCDBoot where he happen to be in a similar situation.

I immediately ran BCDBoot and restarted the PC, changed BOOT Order to Device 0 and it just wont boot properly.

Thinking, BCDBoot had already corrected the BOOT miscue, I thought Windows Repair could do things differently this time. I popped the Windows 7 installer and went on Repair mode and voila … the PC booted normally. Checking on Disk Management, my rig now says:

  • BIOS Boot Order on Device 0
  • Device 0, 1.5TB (System, Boot, Primary Partition, C:\, C:\Windows)
  • Device 1, 1.5TB (Active, Primary Partition)

I then physically removed the faulty hard drive for one last reboot … and everything just are back, all 1.35terabytes of RAW files and some new knowledge on how Windows 7 installs and fixes itself!

Written by totogamboa

August 14, 2011 at 5:25 am

Posted in On General IT Stuff, On Photography, On Rants

Tagged with , , , , , , , , , , , , , , , , , , , ,

Google+ … And What It Means For Photographers Like Me

with 3 comments

First of all, BIG thanks to my friend Rhamille for sending me a Google+ invite.

I started with Flickr, using a Canon A40 point and shoot and remained a paying customer for years to publish my photos when I realize I want more. At first I thought Flickr was all that I wanted. Hi-resolution, un-tampered photo quality, album management, peer feedback, specialized photo groups, and Explore!!. Then I realize my friends, the people that I also want my photos shown, were not there.

Then Facebook came. I am actually a late Facebook adopter as it has been quite sometime before people were able to convince me to take the social networking plunge. I resisted Facebook for some time but as soon as I experienced the power of the “Like”, I never looked back. I still have my Flickr account, which I don’t update anymore since August 2010 and it says, “Hey Toto Gamboa (Not Uploading Pics Here Anymore)! Your Flickr Pro account has expired. Don’t panic! You can only see 200 photos, but the others are safe & sound. You can see them if you renew.”. As if I care if I don’t renew! :P

I have used Facebook’s photos for my photographs since last year but there is so much to be desired from Facebook’s Photos. And there are lots of negative things a photographer can say regarding Facebook’s photos. And you will always hear from everybody that Facebook is, first and foremost, a social networking site and not to be compared with against photography centric sites such as Flickr and the likes. And, did I say that the first note I wrote in Facebook was on how I was so disgusted with its photos? Lols! I had sworn that the moment there is another social networking site that will give photos some importance, I wont hesitate to quickly adapt to it.

And voila! G+! seem to be the answer to my prayers. Not to bore you with what Google+ is, but as a photographer, it’s like Facebook and Flickr rolled into one and then some and A LOT MORE! Here is my quick experience with Google+ Photos:

  • (UPDATE As of Aug 02 2011) Hi-resolution. Great implementation I must say. Documentation says one can upload as large as a 2048 x 2048 image. Though true, you can only view “as is” this size in your browser if your monitor is large enough to contain this big of an image. But since most monitors are way smaller, you cannot. G+ adjusts the size of the display of your image to the size of your browser. The rationale is that, you must be able to view a photo in its entirety without forcing you to scroll horizontally and vertically. However, you can still get the 2048 x 2048 photo by downloading this. There is one quirk with this design though, since most monitors are orientated to landscape, vertically cropped/framed photos are re-sized on display that the result makes them really small. This is done so you can still view the photo in its entirety. Wish G+ have an option to turn resizing off on vertically framed photos.
  • (UPDATE As of July 14 2011) No compression, No Re-sizing. I uploaded a 1280 x 720,  782KB photo, and Google+ retained all its gory and glory. It seems that when you upload photo that is within the limits of Google+ (as long as your browser size is capable of displaying your photo’s full resolution), no compression and resizing is done. Woooohooo!+1.
  • Image linking is HTTPS-based which assures me that my photo won’t be further degraded by any other means. You can check out my other blog on this very issue.
  • Linking with Picasaweb. This means that I can now reach another set of audience for my photos. The photography-centric ones, which I lost, when I stopped using Flickr.
  • EXIF. I don’t have any issues showing the shooting info of my photos to everybody so this can be a plus to me.
  • (UPDATE As of July 20 2011) Auto Language Translation. Ever wonder how to understand when someone post some comments in your photo in Italian or French? Google+ does the translation for you!
  • It’s FREE. No limits. You can upload as many photos as you like as long as you upload via the Google+ interface. If you upload via PicasaWeb, you are only limited to 1GB of space.
  • And my friends, other than those photography-centric folks I interact with, can see and appreciate like they are in Facebook.

Google+ is just 2 weeks old since invites were started. And a lot of things remain to be seen. However, despite being in beta, it looks very promising and most of the things I wanted from Facebook are there.

Of course, for things to get better. All my friends in Facebook need to join Google+ first! :)

Now it’s for you to find out what is in store with Google+ photos! … check out my Google+ albums! :)

Written by totogamboa

July 11, 2011 at 11:26 pm

Posted in On Photography, On Rants

Tagged with , , , , , , ,

Check if your Internet Service Provider Degrades Photo Quality

with 4 comments

Below are two photos linked to a single file in my server. The first one you see should be in its highest quality while the second one should be exactly the same as the first IF YOUR INTERNET SERVICE PROVIDER DOES NOT DEGRADE IMAGE QUALITY by routing the image to a bandwidth optimization server. If you happen to see the second image to be inferior in quality, then the ISP you are using is doing something to save on bandwidth.

You should see this image in its highest quality!

Here is another test reference:

via https (non-degradable)

via http (degradable)

Pay attention to details like the edges of the tree and the bird, also my signature. Check if you can see some pixelized portions and discolorations.

To resolve this issue, host your photos on services that offer Secure Socket Layer (SSL) for your site. This way, your photos will be encrypted while it goes from your host server to your viewer’s browser. This way, ISPs won’t be able to ‘touch’ or degrade your images so they can save on badwidth. So far, this is the only way I can think to circumvent this issue.

Hope this helps!

Written by totogamboa

May 17, 2011 at 3:29 pm

Posted in On Rants

Tagged with , , , , ,

Thoughts on Designing Databases for SQL Azure – Part 3

leave a comment »

In the first article of this series, I raised an issue considered to be one of sharding’s oddities. The issue raised was what would one do should a single tenant occupying a shard exceeds a shard’s capability (e.g. in terms of storage and computing power). The scenario I was referring in the first article was that I opted to choose “country” as my way of defining a tenant (or sharding key). In this iteration, I’ll once again attempt to share my thoughts on how I would approach the situation.

Off the bat, I’d probably blurt out the following when ask how to solve this issue:

  • Increase the size of the shard
  • Increase the computing power of the machine where the shard is situated

On in-premise sharding implementations, throwing in more hardware is easier to accomplish. However, doing the above suggestions when you are using SQL Azure, is easier said than done. Here is why:

  • Microsoft limits SQL Azure’s database sizes to 1GB, 5GB, and 50GB chunks.
  • The computing instance of where a shard can reside in SQL Azure is as finite as well

I have heard of unverified reports that Microsoft allows on a case-to-case basis to increase an SQL Azure’s database size to more than 50GB and probably situate a shard on some fine special rig. This however leads to a question on how much Microsoft allows each and every SQL Azure subscriber to avail of such special treatment. And it could probably cost one a fortune to get things done this way.

However, there are various ways to circumvent on the issue at hand without getting special treatment. One can also do the following:

  • You can tell your tenant not to grow big and consume much computing power (Hey … Flickr does this. :P )
  • You can probably shard a shard. Sometimes, things can really go complicated but anytime of the day, one can chop into pieces a shard. Besides, at this point, you could have probably eaten sharding for breakfast, lunch and dinner.

So how does one shard a shard?

In the first part of this series, I used as an example of sharding a database by Country. To refresh, here is an excerpt from the first article:

Server: ABC
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
1 john file1 grmy
2 john file2 grmy
6 edu file1 grmy
Server: ABC
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy
Server: CDE
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
3 allan file1 can
4 allan file2 can
5 allan file3 can
9 jon file1 can
10 jon file2 can
11 jon file3 can
Server: CDE
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy

In the sample above, the first shard contains data related only to grmy (germany) and the second shard contains data related only to can (canada). To break the shard further into pieces, one needs to find a another candidate key for sharding. If there is none, as in the case of our example, one should create one. We can probably think of splitting up a country by introducing regions from within (e.g. split by provinces, by cities, or by states). In this example, we can probably pick city as a our sharding key. To illustrate how, see the following shards:

Shard #1
Server: ABC1
Database: DB1
Table: userfiles
userfiles_pk User uploaded file country_code city_code
1 John file1 grmy berlin
2 John file2 grmy berlin

 

Shard #2
Server: ABC2
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code city_code
6 edu file1 grmy hamburg

By deciding to further subdivide a country by cities where each city becomes a shard, the following statements would be true:

  • The new sharding key is now city_code.
  • Our shard would only occupy data related to a city.
  • Our shard would only occupy data related to a city.
  • Various shards can be in the same server. Shards don’t need to be in separate servers.
  • The increase in the number of shards would also increase the amount we spend on renting SQL Azure databases. According to Wikipedia, Germany alone have 2062 cities. This is some serious monthly spending that we have here. However this example is just for illustration purposes to convey the idea of sharding. One can always pick/create the most practical and cost-effective key for further sharding to address the issue of going beyond a shard’s capacity without the spending overhead due to poor design choices.
  • At a certain point in the future, we might exceed a shard’s capacity once again breaking our design.

**************************************************
Toto Gamboa is a consultant specializing on databases, Microsoft SQL Server and software development operating in the Philippines. He is currently a member and one of the leaders of Philippine SQL Server Users Group, a Professional Association for SQL Server (PASS) chapter and is one of Microsoft’s MVP for SQL Server in the Philippines. You may reach him by sending an email to totogamboa@gmail.com

Written by totogamboa

March 13, 2011 at 4:09 am

Posted in On Cloud, On Databases, On Software Development

Tagged with , , , , , , , , , , ,

Wild Bird Photography – Oriental Skylark

leave a comment »

One of the most uncommon but conspicuous birds out there in the field is the Oriental Skylark (Alauda gulgula). Conspicuous, in the sense that when it gets excited, it would stand tall and proud with its crest raised despite its minute size,  and would give a very loud distinct shrill. However, you often get to see these birds in photographically boring dry open fields as this species does not perch on bushes or trees. With these circumstances, shooting the bird at an angle higher than its eye level, you will get the boring ground as its backdrop (as shown below).

In the few times I have encountered this species, I would often wish to capture it on camera properly. What I intend to have is to get a creamy bokeh/background. To do this, you can’t shoot from an elevated angle (as shown above), unless the bird is on an edge of an elevated mass that would eliminate the ground as the backdrop or would make the ground far enough not to be included in your thin depth of field. Instead, you need to be down on all four. This means you need to be on a prone position to achieve the effect (as shown below). This enables you to avoid shooting the ground as the bird’s backdrop.

Shooting the bird on the ground on a prone position gives one an eye level shot with both the foreground and background seem to merge and melt leaving the subject greatly emphasized. Below is a sample of this effect.

Shooting Disclosure

  • Gears:
    Canon 50D, EF 400mm f5.6L, 2-Pound Rice Bag
  • Settings:
    Shot @ f5.6, 1/640″, ISO320, Spot Metering, Auto White Balance, Aperture Priority, Cropped 16:9 to 3.6MP, RAW, Handheld, Prone Position
  • Lighting
    8:36am Light, Overcast
  • Others:
    Some very minimal sharpening and color vibrancy adjustments in Photoshop

Written by totogamboa

February 27, 2011 at 9:14 am

Posted in On Photography

Tagged with , , , , , , , , , , , , ,

Wild Bird Photography – Dollarbird

with 4 comments

In wild bird photography, gears and skills do really matter. Both needs to go hand in hand to produce the best snapshot of a beautiful avian subject. In the Philippines, it is often that you and your gears will be tested. For decades, our subjects here are hunted everywhere that birds here often avoid human encounters as much as possible. This cautiousness adds to the fun and challenge to wild bird photography in the country. Because of this, 400mm lenses are considered short. You have to exert more effort in terms of executing the shot to produce decent photographic captures at par with those that have bigger, longer and generally better equipment. However, one cannot fret with what he has. Working a bit more with what you have does the trick and solve some of the problems.

Compensate

Having known that I have a short lens at only 400mm (Canon EF 400mm f5.6L) with a fairly slow speed (f5.6) and a camera that is known to produce noisy images at high ISOs, I need to do a lot more to compensate on the limitations I have. In this article, I would detail how using a 2X teleconverter on a 400/f5.6 lens mounted on a Canon EOS 50D to shoot a very difficult scene can still be accomplished with satisfactory results. Here is how:

Below is a photo of an uncropped 800mm shot of a Dollarbird (Eurystomus orientalis) from more than 30 meters. I dont normally use a 2X teleconverter but I was tempted to because the bird perched motionless for minutes after I got several shots with just the bare 400mm lens mounted on my cam and the sheer distance between me and the bird. Getting nearer is also impossible.

Here is a cropped version of the same image. I cropped the photo down to 2.8 megapixel.

Shooting Disclosure

  • Gears:
    Canon 50D, EF 400mm f5.6L, Kenko Pro-DG 2X, Manfrotto 755X + Gimbal Head, 2-Pound Rice Bag, Remote Shutter 
  • Settings:
    Shot @ 800mm, f16, 1/40″, ISO320, Evaluative Metering, Auto White Balance, Full Manual, Cropped 16:9 to 2.8MP, RAW, Liveview, Remote Shutter
  • Lighting
    8:42am Light, Overcast
  • Others:
    Some very minimal sharpening and color vibrancy adjustments in Photoshop

Often, we hear discouraging comments on the use of 2X teleconverters. It definitely degrades image quality even when use with large aperture wildlife lenses such as those with f2.8′s and f4′s. Using a 2X on an f5.6 lens would surely raise eyebrowes. But when you are limited to shoot with what you got, and in my case, I only have a Canon EF-400mm f5.6L, one needs to do a lot of compensating to get decent output from lowly setup with a 2X. And a couple of requirements to effectively know how to compensate is you need to know how a photograph is made and you know very well your gears’ capability and limitation.

In the above photo of the Dollarbird, despite the constraints I had during the time of the shoot, I still managed to get a decent shot. Here are the key ingredients in executing this shot:

  • LiveView. Knowing that using a 2X on an f5.6 will force you to go full manual, using LiveView is one very effective technique. But of course this can only apply since the Dollarbird lingered long enough for me to set things up. Using LiveView in this scenario, one would get AUTOFOCUS using contrast detection method. My Canon EOS 50D allowed me to do this. Some cameras would probably do the same. Using LiveView in this scenario, it also allows you to visually zoom in to 10X using your LCD to get better confirmation if you have focused well on the subject. In the 50D, you can have these features work for you. You get to zoom in to your subject and get aufo-focus.
  • 2-Pound Rice Bag. At 800mm, very minute shaking is very visible. By increasing your LCD view to 10X (via LiveView), not only is the shake visible, it can make you dizzy :P . Putting weight on your rig would dampen the effects of this shake. It also speeds up in stabilizing your rig so you get to shoot in the soonest possible time. In my case, I have this useful 2 pound weight functioning as a poor man’s image stabilizer. All I have to do is place the weights on top of the lens where its center of balance is.
  • Remote Shutter. Without a remote shutter. This scene is hard to execute. One can use the cam’s timer though but that is cumbersome.

The above key ingredients helped in allowing me to capture this scene. Though it won’t surpass the quality of a shot using a bare lens at the same focal lenght, the result is decent enough to merit a space on my harddrive. Without any one of the three, it would be very hard to get a decent output from this scene. Compensating can do wonders especially for photographers that don’t have those desirable longer and faster lenses and better camera bodies. The same techniques used here can be applied using better gears of course. :)

Written by totogamboa

February 18, 2011 at 6:42 pm

Posted in On Photography

Tagged with , , , , , , , , , , ,

Thoughts on Designing Databases for SQL Azure – Part 2

leave a comment »

In the first article, I showed an example how a database’s design could impact us technically and financially. And sharding isn’t all just about splitting up data. It also brings to the table a group of terrible monsters to slay. There are a lot of concerns that needs to be considered when one attempts to shard a database, especially in SQL Azure.

NoSQL, NoRel, NoACID

In breaking things apart, one is bordering on clashing religions. One monster to slay is the issue of ACIDity. People discuss NoSQL, NoRel, NoACID to be one of the trends out there. And most even swear to the fact that these approaches are better than SQL. In my case, I prefer to call it NoACID and it is not by any means more or less than SQL. I have NoACID implementations on some projects I had. And I love SQL.  To simplify, I’ll put in these trends in a NoX lump as they commonly attempt to disengage with the realities of SQL.

For me, NoX is not a religion, it is simply a requirement. The nature of the app you build will dictate if you need to comply to the principles of ACID (Atomicity, Consistency, Isolation, Durability). If ACID is required, it is required regardless of your data and storage engine or your prefered religion. If it is required, you have to support it. Most cloud apps that we see, like Google and Facebook, could probably have ACID to be absent in their requirements list. Google is primarily read only so it does make sense to have data scattered all over various servers in all continents without the need for ACID. By nature, ACID in this regard, can be very minimal or absent. Facebook on the otherhand is read/write intensive. Seems like it is driven by a massive highly sophisticated message queuing engine. Would ACID be required in Facebook? I am not quite sure about Facebook’s implementation but the way I look at it, ACID can be optional. ACID can well be present in operations concerned only to one tenant in case of an FB account. Outside of this, the absence of ACID could probably be compensated by queuing and data synching.

If Facebook and Google decided to require ACID, they could be facing concerns on locking a lot of things. While locked on, latency could be one of the consequences. It is therefore very important to lay out firsthand if ACID is a requirement or not. For a heavy transactional system, a sharded design presents a lot of obstacles to hurdle. In SQL Azure, this is even harder as SQL Azure does not support distributed transactions like we used to with SQL Server. This means, if your transaction spans across multiple shards, there is no simple way to do it as SQL Azure does not support it, thus ACID can be compromised. SQL Azure however does support local transactions. This means you can definitely perform ACIDic operations within a shard.

To be continued…

**************************************************
Toto Gamboa is a consultant specializing on databases, Microsoft SQL Server and software development operating in the Philippines. He is currently a member and one of the leaders of Philippine SQL Server Users Group, a Professional Association for SQL Server (PASS) chapter and is one of Microsoft’s MVP for SQL Server in the Philippines. You may reach him by sending an email to totogamboa@gmail.com

Written by totogamboa

February 11, 2011 at 3:36 pm

Posted in On Cloud, On Databases, On Software Development

Tagged with , , , , , , , , , , ,

Wild Bird Photography in the Philippines – Scaly Breasted Munia

with one comment

One of the most common beautiful bird you will encounter in the Philippines is the Scaly-breasted Munia (Lonchura punctulata). It can almost be found anywhere in the Philippines. This species can also be one of the easiest to photograph considering that it is often close to human habitation. Where there are rice fields, you are guaranteed to see this bird.

Below is a photo I got in one of our sorties in San Juan, Batangas. I was patiently waiting for waders on a nearby pond when a few meters from where I hide, this fellow showed up. It probably got curious by my presence that it stayed a while and allowed me to get a bit closer and have some shots that I like.

Shooting Disclosure

  • Gears:
    Canon 50D, EF 400mm f5.6L, Manfrotto 755X + Gimbal Head
  • Settings:
    Shot @ 400mm, f5.6, 1/50″, ISO100, Spot-Metering, Auto White Balance, Aperture-Priority, Cropped 16:9 to 3.2MP, RAW
  • Others:
    Some very minimal sharpening and color vibrancy adjustments in Photoshop

On a mid-day harsh light, I usually go ISO100 to avoid getting a very fast shutter speed so more light gets captured with the sensor shutter being opened much longer. With ISO100, I also avoid some unwanted noise and it enhances the creaminess of background blur. Though it is hard to pull off and increasing the risk of getting a blurred shot, I prefer to shoot around 1/40 secs to 1/200 secs as I almost always get better color in this range (provided I am on a sturdy tripod). I just don’t know the technical reason but I suspect, the more time I allow the camera to absorb light, the better the output I got. This is why I always try to bring down ISO to as much as I can for as long as the shutter speed is within my prefered range. By constant practice, blurring cause by shake and slow shutter speed can be avoided. The bird allowed me to focus on its eye as it gave me a nice glance as shown in the photo. I also got lucky that it perched on a really photogenic decaying branch with a good greeny background from the distance.

The bird gave me a few shots but the above photo is the one I like most. It flew the moment I attempted to get closer.

Written by totogamboa

February 10, 2011 at 9:30 am

Posted in On Photography

Tagged with , , , , , , ,

Wild Bird Photography in the Philippines – Part 3

leave a comment »

This is so far a 3 part series of what wild bird photography is to me. I would probably evolve this series over time to make it current as much as possible.
Wild Bird Photography in the Philippines – Part 1
Wild Bird Photography in the Philippines – Part 2

Previously,  in part 1,  I discussed how I came to photographing birds and detailed what you need to get a good start. In Part 2, I gave some ideas where one can usually find birds. In this article, I’ll give an idea how a bird photographer goes out there in the wilderness to look and start capturing beautiful images of birds.

A Birdnut’s Sortie

Often, when bird photographers (or birdnuts) go out, it is simply referred to as a sortie. A sortie is basically a mission to go out and photograph birds. Sorties vary from just a few hours birding away from home to weeks-long missions. Sorties can be done in singles or in groups. Sorties can be nearby, or one needs to travel for hours. Some sorties go from one country to another. Most sorties are done on foot, but there are sorties where birdnuts are in the comfort of their cars or boats. Some bird photography sorties involve laying out an elaborate plan compose of preparing the things needed for the trip, plotting destinations, identifying the risk and dangers associated with it and a lot of things go with it.


Reserva, Baler Sortie with Wild Bird Club of the Philippines

But How Do I Go About My Own Bird Photography Sorties?

I am a weekend warrior.  Though there were times when I went out on a whim, I mostly do my sorties on a weekend. I usually plan days ahead of the trip. I would typically go with a birding buddy or with a group for safety reasons. But I have gone out alone.

I usually prepare all the logistics needed and my gears a day before the trip. I’d check the vehicle for its condition and make sure it is ready to go. I’d have all the batteries recharged. I’d have the GPS properly loaded with routes and waypoints. Then I’d go to the nearby grocery or convenience store for my supply of food and drinks. Here is the usual stuff that I buy and prepare:

  • bottled water (Absolute) and some flavored drinks (Gatorade’s Propel) for rehydration
  • some biscuits (Skyflakes or some other brand) just to fill my tummy when out in the field or when on a long drive
  • some sweets (chocolate bars) to give me some caloric/energy boost when things go tough in the field
  • first aid kit
  • clothing (hat, extra shirts)
  • birding gears

I’d make it a point to get enough sleep but often I do get excited that I can’t sleep properly hours before the trip. Typically, I would wake up early in the morning (e.g. 4:00am for Candaba, 3:00am for Subic, 5:30am if it is just nearby), get some quick shower, then off I go. I usually make a quick stop in the nearby 7-11 store to buy me some packed hot meals for breakfast (I love their liempo on plain rice) and some bags of ice to fill my cooler, before I hit some more roads or pick up my birding buddy.

Once  on the site, it is usually breakfast time for me. :) Then after that .. it is mostly birding, birding, birding! Sometimes I don’t get to rest, sometimes I don’t get to eat when in the field. What my day is like will all depend on how many and what kinds of birds I encounter. Then I go home!

My Most Memorable Birding Sorties

For more than a couple of years now, I have been to several birding places all over the country. My most memorable was getting stranded in Subic in 2009 while Metro Manila was being flooded by a record breaking storm Ondoy (Typhoon Ketsana). My birding buddy Dennis and I were just clueless what was about to happen during that day. The next thing we knew, Metro Manila had its most catastrophic flooding while we got stuck somewhere in Subic and left no other choice but to wait out until the storm passes through. Another most memorable birding trip I had was when I had gone with by fellow birdnuts in Mindoro. We had a chance to capture in photograph the rare Scarlet-collared Flowerpecker (Dicaeum retrocinctum).


Mindoro Sortie with Bulbuleros 400

The Risks

  • The hardest part is when going home. Often, you are so tired to move and drive yourself home. When I am with a birding buddy, we usually alternate on the wheels to lessen the risk of getting so sleepy on the road while driving for home. If sleepiness is unbearable, I/we stop to take a nap.
  • Insect bites. I remember a birding buddy of mine (Dennis), got hospitalized due to insect bites, after we went to the forest of Subic.
  • Wild bee stings
  • Snake bites
  • Getting attacked by wild animals
  • Getting stranded
  • Encounters with some not so really nice people (thieves, illegal loggers, etc). I guess the worst thing that will happen to you is when you get kidnapped by known terrorist groups and you would have to save your life as what happened to a fellow bird photographer. I happen to cross paths and shoot birds with Ivan personally one weekend in La Mesa Ecopark. Such a great guy by the way.
  • Weird accidents
  • Being flown off by a giant jungle woodpecker and be brought to its hole/nest on top of a 500 meter tree

The Lows

  • Going home empty handed
  • Going home with a ruined camera or a broken lens
  • Getting to the birding site only to find out you left your birding lens

The Highs

  • Getting stranded
  • Getting a rare photo lifer. In bird photography lingo, a lifer means you got to photograph a bird species for the first time
  • Getting a close up of a very beautiful bird
  • Getting a good photo of a target bird
  • Bump in the field with fellow birdnuts
  • Lastly, getting attacked by a thousand sunbirds

Wild Bird Photography in the Philippines – Part 1
Wild Bird Photography in the Philippines – Part 2

To see my photos of Wild Birds of the Philippines, check my album here.

Written by totogamboa

February 10, 2011 at 1:16 am

Posted in On Photography

Tagged with , , , , , , ,

Thoughts on Designing Databases for SQL Azure

with 4 comments

Cloud has been the new darling in the world of computing and Microsoft has provided once again developers a new compeling platform to build their new generation applications on. In this picture enters SQL Azure, a cloud-based edition of Microsoft’s leading database product, Microsoft SQL Server. One would ask how one designs an RDBMS database for the cloud and specifically for the Azure platform. It is basically easy to create a simple database in SQL Azure but database creation and database design are two different life forms. So we ask, how are databases be in the cloud and in SQL Azure? In this blog, I hope to provide an answer.

Scale Out

First and foremost, the cloud has been one of the primary solutions to consider when an application is in the realm of a massive scalability issue. Or to put it in another perspective, the Cloud was both a catalyst of the emergence of extremely large applications and a result of the webification of a lot of things. Design considerations need to factor in unheard of scales that can span from almost everywhere. Bandwidth needed is of the global scale, hard disk space are in the petabytes, and users are in the millions. When you talk about building an application to handle this scale, one can’t do anymore the usual stuff any one of us used to do. One now needs to think massively in terms of design. E.g. No single piece of hardware can handle the kind of scale I just mentioned. So how does one design a simple database that is intended to scale up to the degree that I just mentioned?

The Problem

Supposing, we have a great idea of developing the next big thing and we named it FreeDiskSpace. FreeDiskSpace allows us to upload massive amounts of files where files can be anything one can think of. The scale of this next big thing is thought to be in the range of having 500 million users scattered around the globe with each users allowed to upload an unlimited amount of files.

The Challenge

With such order of magnitude, the many challenges we will be facing will definitely lead and force us to question every bit of our knowledge and development capability that we might have been using when we are used to doing gigs in the few GBytes of databases with a few thousand users. If what is allowed makes each user upload files in the range of not less than 5GB, one would need to store 2,500 petabytes worth of files. How does one store 2,500 petabytes worth of files and manage the uploading/downloading of 500 million users?

In the Cloud, you are less likely to see a bunch of supercomputers handling this kind of scale. Rather, you are more likely to see hundreds of thousands of commodity servers woven together forming a cohesive unit. Making use of these mesh to behave like it is as if it is operating as a unit requires design patterns that are more appropriate for these environments.

Database Sharding in SQL Azure

First, what is database sharding. To put it simply, database sharding is a design pattern to split up data into shards or fragments where each shard can be run on its own on a separate instance running on a separate server located somewhere. Each shard contains the same structure as that of another but contains different subsets of data.

Current version of SQL Azure offers NO feature that handles automatic scaling. This will surely change in the near future as everybody doing applications handling these massive loads will somehow need an automatic way to allow the application to handle this amount of load and data easily.  One way that is guaranteed to be commonly used will be sharding databases. However for now, sharding in SQL Azure isn’t automatic. If one needs to shard, one needs to do it by hand. As I have said, in the future, Microsoft will be compelled to give us an easy way to practice sharding. Hopefully, that future is near.

Now we would ask, why do we need to shard databases in SQL Azure. The cloud and specifically with SQL Azure, with its limitations, forces us to use this design pattern. SQL Azure only allows us 50GB for a database’s maximum size on a virtual instance running on an off the  shelf commodity server. With that, we surely need a way to work around these constraints if we see our data to grow for like 500 terabytes. Second, if we foresee a huge number of users hitting our applications, doing this on a single piece of hardware would spawn concurrency nightmares that would choke our databases to death.

Though there are other approaches to solve our scaling challenges, there are other interesting motivating factors where sharding becomes a very compelling option. Firstly, it is more likely for a database server to process small databases faster than larger ones. Secondly, by utilizing multiple instances of computing power, we can spread processing load across these instances to more processes can be executed all at once.

How To Shard

Let us try to go back with our FreeDiskSpace killer app (assuming we provide each user up 1GB in database space) and practice some very simple sharding so we have a partial understanding of what sharding is all about. If we are to design the database using a single instance, the table would look like this:

Server: ABC
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
1 john file1 grmy
2 john file2 grmy
3 allan file1 can
4 allan file2 can
5 allan file3 can
6 edu file1 grmy
7 roman file1 ity
8 roman file2 ity
9 jon file1 can
10 jon file2 can
11 jon file3 can
Server: ABC
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy

If we are to shard, we need to decide on how we split up our database into logical, manageable fragments. We then decide what data is going to occupy each fragment (we can probably call the occupier our “tenant”). Based on tables we see above, there are 2 fields that we can potentially use as keys for sharding. These fields are user and country. These fields should provide answers to a question: How to we sensibly split up our database? Do we have to split it up by user, or by country? Both are actually qualified to be used as basis for splitting up our database. We can investigate each though.

Sharding by field: User

Server: ABC
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
1 john file1 grmy
2 john file2 grmy
Server: ABC
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy
Server: CDE
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
3 allan file1 can
4 allan file2 can
5 allan file3 can
Server: CDE
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy

By deciding to make each user as a tenant in our database, the following statements would be true:

  • Our shard would only occupy data related to a single user.
  • Various shards can be in the same server. Shards don’t need to be in separate servers.
  • It would render country table a bit useless and awkward. But we can always decide to get rid of it and just maintain a column indicating a user’s country of origin.
  • Now with a one-user-one-database structure, 1 million users mean 1 million separate databases. In SQL Azure’s terms, assuming a 1GB costs us 10$ a month, designing our shards by user means we have to spend $10 million monthly. Now that is no joke!

Sharding by field: Country

Server: ABC
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
1 john file1 grmy
2 john file2 grmy
6 edu file1 grmy
Server: ABC
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy
Server: CDE
Database: DB1
Table: userfiles
userfiles_pk user uploaded file country_code
3 allan file1 can
4 allan file2 can
5 allan file3 can
9 jon file1 can
10 jon file2 can
11 jon file3 can
Server: CDE
Database: DB1
Table: country
country_code country
grmy germany
can canada
ity italy

By deciding to make country as a tenant for each of our database, the following statements would be true:

  • Our shard would only occupy data related to a country.
  • Various shards can be in the same server. Shards don’t need to be in separate servers.
  • It would render country table unnecessary as each database represent a country. But we can always decide to get rid of this table.
  • Asnwer.com says there are only 196 countries in the world. With a one-country-one-database structure,  this means,  if we are getting the larger 10GB, 100$ a month SQL Azure databases, we only need to spend $19,600 a month.
  • This design decision though is pretty useful until we ran out of space in our 10GB design to contain all users related to a country. There are solutions to this problem though. However, I will try to discuss this in my future blogs.

It is pretty clear that picking the right key for sharding has obvious ramifications. It is very important for us to determine which ones are appropriate for our needs. It is obvious that picking country is the most cost-effective in this scenario while picking user could shield you much longer on the 10GB limit problem.

This article only presents partial concepts of sharding but nevertheless has given one the very basic idea of how database sharding can be applied, and how all these translate when you choose SQL Azure as a platform.

Thoughts on Designing Databases for SQL Azure – Part 2

**************************************************
Toto Gamboa is a consultant specializing on databases, Microsoft SQL Server and software development operating in the Philippines. He is currently a member and one of the leaders of Philippine SQL Server Users Group, a Professional Association for SQL Server (PASS) chapter and is one of Microsoft’s MVP for SQL Server in the Philippines. You may reach him by sending an email to totogamboa@gmail.com

Written by totogamboa

January 31, 2011 at 7:42 am

Posted in On Cloud, On Databases, On Software Development

Tagged with , , , ,

« Older Entries
Newer Entries »
Follow

Get every new post delivered to your Inbox.