Part of the difficulty of living in a large city (Columbus is the 15th largest in the nation, just below San Francisco) is that the ecological freedoms possible in rural life are much more limited. To the extent that houses, doubles and apartments in the Short North have backyards at all, they are very small and much more tightly packed together than in either the country or the suburban wastes of Upper Arlington and Westerville. This means that composting must occur in a very small and confined space, not in a free-spirited pile.
Two options present themselves: Worm Composting (Vermiculture) or Bin Composting. Worm composting can be carried out indoors in any space fitting a tote bag. The downside is that if worm composting gets out of control (odors and fruit flies being the two serious contenders), you're going to have an unpleasant time of it. Bin composting requires at least a little bit of yard space, but if you've got it, all the decomposition goes on outside, leaving your interior without the grungies and whatever yard you've got rich and organic.
If you're looking for a worm compost startup unit, I recommend
Urban Gardener, 940 N. High Street. They've got a multitiered set of stacking bins for separating "castings" (come on, it's worm poo) from your vegetables, a drain for "compost tea" (come on, it's worm piss), coconut bedding to give the worms an initial home, and an inside deal with some on-demand harvesters of red wigglers in nearby Zanesville, which is a much better option than getting your worms shipped half-dead from Texas.
If, on the other hand, you're looking for that stylish, understated and much easier-to-handle composting bin, try Backyard Experience at 3515 North High Street in Clintonville. They've got two fairly large black bins and a tumbler unit for sale at a reasonable price, along with some unnecessary "compost bio-starter kit." Unless you're composting on a concrete slab in a brownfield, the little buggies and germies that will do your composting will come along on their own.
These places are not your uncle's Agway. Most of Urban Gardener's stock tends toward the frou-frou, with mystic fairy windchimes and stepping stones with empowering 19th century feminist literary quotes. Backyard Experience offers the Ohio State University Buckeye Garden Gnome. But when it comes to composting supplies for small spaces, together they do the job pretty well... and they're small locally-owned businesses to boot, which will give you that crunchy, whole-grain, ethical kind of feeling when you head out the door with your goodies.