Sorry for the delay! This is fine, so long as it's styled as the last act after everybody else has left anyway. And as you said, the impact will be pretty minor--it's a very well-defended fortress and the window will be quite slim between revealing their presence and location by starting to shoot people and facing a counter-attack with overwhelming force, including magical ranged attacks on them and the griffons. A completely clean getaway is unlikely but we leave up to you what that means.