Mr Hart’s letter said that although the decision to use the camp to house asylum seekers had caused “much frustration and anger” in the community, the Home Office claimed it had “little option at the time.”
The first of around 250 male asylum seekers aged between 18-35 moved into the site in September of last year.
The decision comes after an independent inspection of the Penally site, as well that of Napier Barracks in Folkestone, Kent, found they were “impoverished, run-down and unsuitable for long-term accommodation”.
Mr Hart’s letter on Tuesday said: “I am very pleased to confirm that the Home Office has agreed to return Penally Camp to the Ministry of Defence by March 21.
“The decision has been taken following many weeks of discussions between myself, the Wales Office and the Home Office ministerial team.
“During those meetings and discussions (which have been taking place almost daily over the last few months) we have tried to ensure that the concerns of everybody involved have been properly and legally accounted for.
“The impact of flight and accommodation regulations due to Covid-19 have made this much more complicated than would usually be the case.
“I am deeply conscious that the manner in which the use of Penally came about caused much frustration and anger. The Home Office has recognised this contributed to heightened tensions, but stress they had little option at the time.”
Mr Hart’s letter paid tribute to police, the health board and the local authority for their efforts in “maintaining a measured approach to a situation thrust upon them at short notice”.
The Home Office has faced repeated criticism over its use of Penally Camp and Napier Barracks to accommodate asylum seekers.
However, Home Secretary Priti Patel and Immigration Minister Chris Philp have both previously defended the use of such sites.
Last week, an independent report found “fundamental failures” over housing asylum seekers at the military barracks, parts of which were “filthy”.
It followed inspections by the Independent Chief Inspector of Borders and Immigration (ICIBI) and Her Majesty’s Inspectorate of Prisons (HMIP).
Inspectors said: “The environment at both sites, especially Napier, was impoverished, run-down and unsuitable for long-term accommodation.”
At both locations, residents described feeling trapped in poor conditions and feared that if they moved out they would jeopardise their only source of support and possibly their asylum cases.
Residents at both locations reported being shouted at and intimidated by protesters and members of the public who did not want them there.
Penally Camp is used by the Ministry of Defence for training courses, which have been rescheduled or moved to alternative military sites during their use as temporary accommodation.
Police and Crime Commissioner for Dyfed-Powys Police, Dafydd Llywelyn, said: “I’m extremely pleased to hear today’s announcement, and that finally we will see the closure of the asylum centre in Penally.
“It will be a welcomed relief not only to local residents in Penally and the surrounding area, but also to the asylum seekers within the centre.”
He said a lack of strategic planning, as well as a lack of community engagement, led to “unnecessary pressure being put on local resources at a time when we are trying to protect our communities from a global pandemic”.
Kolbassia Haoussou, director of survivor empowerment at Freedom from Torture, called on the Government to also close Napier Barracks, adding: “How people are housed is about more than just providing shelter. It is a reflection of how we, as a society, treat those who are most vulnerable.
“If this Government is serious about improving the asylum system, it will close the barracks, make quick and fair asylum decisions, and house people humanely within our communities so they can start to rebuild their lives.”