Mercury would be a challenging planet to terraform (though not as challenging as Venus) It has many features that are positive for terraforming, including a low surface gravity (about the same as Mars' [0.38 G]), an internal dynamo that produces a magnetic field (which is especially important as a radiation shield so close to the Sun) and considerable stores of water-ice and organic compounds in its polar craters.
Initial human habitats could be built mostly underground, to allow the establishment of a continuous human presence on the planet before terraforming. However, the above-ground parts of these colonies would need to be rigorously self-contained and pressurized, like a space station. This is because Mercury's surface pressure is close to zero, as a result of its low gravity and proximity to the Sun.
If it were determined that fully terraforming Mercury would be uneconomical or technologically impractical, the other option would be building enormous, transparent, geodesic domes across the planet's surface and terraforming the space contained inside them (as proposed by Wikipedia). This would mirror a method proposed to allow terraforming of Earth's Moon.
Fully terraforming Mercury would require first shielding the planet from excessive solar radiation, before beginning the construction of an atmosphere on the planet. Mercury's native magnetic field is 1/100th the strength of Earth's. This is small; however, if only taking into account Mercury's low gravity and magnetic field, Mercury should have an atmosphere comparable in thickness to Mars. However, Mercury's close proximity to the Sun means that it experiences a much more intense pressure of solar wind, three times greater than that experienced by the Earth. The intense solar wind has prevented Mercury from accumulating an atmosphere thick enough to be relevant to humans.
Therefore, construction of an atmosphere would only be possible once a shielding unit (such as solar shades) has directed some of the solar radiation pressure away from the planet's surface. There is a chance that gases from the solar wind would begin accumulating to form a thin of atmosphere around Mercury; but human terraformers would want to thicken that atmosphere to make the surface of the planet more comfortable for colonists.
Unlike Mars, which contains frozen gases locked up in rocks across its surface, the gaseous components of Mercury's rocks have long been baked away. Gases to build an atmosphere would have to be brought to Mercury from elsewhere in the Solar System. Since excess heat is an significant problem for a planet so close to the Sun, greenhouse gases like methane or carbon dioxide would have to be kept to a bare minimum. Nitrogen for Mercury's new atmosphere could come from Titan. Icy comets could be redirected to Mercury to flood craters with water to create oceans. A supply of oxygen would be needed as well, which could come from the hydrogen and oxygen from the water.
Another problem with terraforming Mercury is that the length of its solar day is about 4223 hours, or the equivalent of 176 Earth days. Currently, temperatures on Mercury differ dramatically between the day side and the night side of the planet. Research has shown that an atmosphere thick enough to sustain humans might serve to even out the planet's temperatures by circulating air; however, Earth plants would still have a very difficult time adapting to such an extreme solar day. Thus, Earth plants might need to be genetically modified to cope with the long nights.
As of right now, it is technologically unfeasible to alter the rotational speed of a planet and this will continue to be impossible for the foreseeable future. One possible workaround to this would be to expand Mercury's solar shielding into a band of mirrors encircling the planet. This would enable the Sun's light to be redirected to light one side of the planet's surface, simulating a shorter day/night cycle than would actually exist.
If terraformed, Mercury will face the problems of a Mountain Planet.