angularjs - Angular UI Bootstrap - Cannot delete popover when triggering element is removed from DOM

I am using a javascript library that will add/remove elements to the DOM when some data is updated.
To add a new element to the DOM it calls a template function returning an element. In my case, the template function is defined inside an angular directive and returns something like return $compile(html)(scope)[0]; because I need to use the UI Bootstrap Popover directive inside the added element.
For the Popover, I need to use popover-append-to-body="true".

My problem is, if the triggering element is removed from the DOM, the popover is never removed. So if we add a new triggering element, a second popover will be appended to body, etc.

Here is a simplified example : http://plnkr.co/edit/AFsXBcaLBAs0F2garbAu?p=preview
Clicking on the "Click" button opens the popover, clicking "Remove" removes the "Click" button, clicking "Add" re-adds the "Click" button and clicking "Click" again adds a second popover to the DOM.

How can I remove the Popover directive when the triggering element is removed from the DOM ?
I need to totally deletes it, not only hide it/remove it from the DOM (I can hide it with popover-is-open but when this is set back to true, I see the popover still exists).
Is there a way to call destroy on the Popover directive of the element that will be deleted ?