Why metal feels colder than wood in the same room

A metal door handle and the wooden door are in the same room all night. In the morning the handle feels much colder, even though a thermometer would probably give a similar temperature for both. My guess is metal transfers heat away from my hand faster, so my skin interprets that as ‘colder.’ Is that essentially the whole story? And would the reverse happen if both materials were warmer than my hand?