Once an alert is triggered its status appears as such and it remains present in the Alert Manager only as a matter of record, no actions will be taken if the limit is reached a second time.
To remove a triggered or active alert select the its row header under the "Status" column and use the Alert->Delete menu or the toolbar button with a cross symbol.