It is a worry. And part of the reason why the US is so actively engaged with Pakistan is because Pakistan does have a nuclear arsenal and that nuclear arsenal could potentially be under threat if Pakistan ever does become a failed state. Although I say Pakistan is not currently a failed state now, it could happen one day.
"The Pakistanis have assured the US that their nuclear arsenal is safe. But many very sensitive sites in Pakistan have been targeted by militant groups."
Pakistan has an extraordinary nuclear arsenal and it’s also involved in a very bitter conflict with its neighbour and rival India; a conflict which led to the creation of its nuclear program. The Pakistanis have assured the US that their nuclear arsenal is safe. But many very sensitive sites in Pakistan have been targeted by militant groups: the general headquarters of the military has been targeted, for examples. And while I was living in Pakistan, there was a site that was targeted.
Although the military never admitted it, it was understood that there were some nuclear components that were stored at this military site. So it's a concern for the West and that is perhaps why part of the reason why the West continues to keep Pakistan as an ally as much as possible.
Photo credit: Telstar