strange soot in skylights

Where did the contractor install the flashing and caulk? That should have worked, if done in the right places. How old is your skylight? If you have a Victorian-era building, the soot is most likely a century’s worth of grime that accumulated in the crawl-space between the ceilings and the roof joists. The grime is usually dust mixed with an oily air-borne residue; either from creosote-smoke that escaped from a leaky chimney, or pollutants that got in from outside and settled (coal smoke, car exhaust, etc). It most likely isn’t the rain that causes it to appear, but the winds that accompany the rain. If the air in the crawl-space gets slightly pressurized the grime will blow out through whatever way the air escapes; in your case the skylight.
Vacuuming-out your crawlspace is impossible, but if you caulk all of the seams around the skylight, both from above and below, it should do the trick.