My partner and I are currently in the same situation (but about a step and a half ahead).

The city will bring gas from the street to your house for free (you will have to hook it up within 6 months or something or they charge you a /month fee).

The city has loan programs for converting oil to gas. The catch is that the savings on the more efficient furnace have to make up for cost of the loan within 10 years (which you will easily be able to do unless you are replacing the world's most efficient oil furnace with the world's crappiest gas furnace).

Other than that, I don't have a lot of specifics on the details (my support of the new furnace project involves moving my piles of my Halloween crap out of the basement). I'll see if I can get more details for you later.

We converted from oil to gas about four years. We bought a Buderus ultra-high efficiency boiler. At the same time we changed all the original storm windows on our 1924 house to Mon-Ray high performance storm windows with e-coating, etc. Both changes were expensive and we took out a loan to make them, despite figuring we might never recoup our costs, because it was the "right thing to do" (we were consuming an unconscionable amount of energy for just two people). Instead, we started saving money the very first month. In other words, our monthly energy bills were less, even with the loan payment included. Plus we were consuming a fraction of the energy. So my advice would be do it. Do it before another winter arrives, even if you have to take out a loan, it's hard for me to imagine that will ever regret it. 
P.S. One great advantage of the Buderus is that it makes the hot water for your house with passive heat captured from the boiler. In other words, no more water heater and virtually free hot water in the winter!

We took care of the getting-rid-of-leftover-oil issue by just burning all of ours beforehand.  It turned out that the gauge on the tank was a little optimistic, so that hoped-for moment of tank-emptying actually came about a week before the install date for our gas furnace (which happened to be in an unseasonably cold late November a few years ago).  I would kind of recommend against an exact repeat of that scenario [I've been told since then that diesel fuel is a reasonable stopgap substitute, but it would have been a lot more helpful to know that back then, hrmph].

Actually, I recently made the switch from gas to oil after learning that I own an oil company.



Thank you Energy for Tomorrow, for pointing out my vested interest!





Now, about that tax break...

@BadCat, as I mentioned above, we got ours removed by offering the guys who removed it the oil that was in it. In fact, they paid us in addition to cutting the tank up and getting it out of there. I'm pretty sure we found them by posting on Craigslist, but maybe our contractor found them for us. I'll ask him.

Crap on a stick! We were all set to get the furnace loan program, but now we're out of luck!
It's complicated financials that I don't really understand, but it has something to do with our home value not being enough and the amount we owe too much (thanks housing market!).
So now we have a near-dead furnace that needs to be replaced before fall, and all of our spare cash is going towards the roof project already underway.
Does anyone know of any other programs that could assist? I can't think we're the only people out there who don't have a lot of equity in their undervalued home, and earn too much to qualify for most of the assistance programs. Thanks!!!

Yep, that's the program we were hoping to get. They were really nice and helpful, but our numbers just didn't work out (recently purchased home, not much equity, declining home value = no go in their spreadsheet). :(