Loki is not Tony’s stuff. Loki is not SHIELD’s stuff either. He’s their concern, sure…but he’s not their ‘stuff.’ Loki will always and forever be Thor’s ‘stuff.’ Loki is Thor’s responsibility and no one else’s. Thor sacrifices the most for Loki. Thor is one of the few that still cares for Loki. Thor just wants to bring Loki home and repair the damage that has been done. He can’t do that now as Loki has become a problem for Midgard. Thor can’t exactly take him away from this now and hearing Loki being called another person’s ‘stuff’ is jarring. It says ‘look, I have what once was yours and what is still important to you and you’re not getting it back easily.’ 

You can call Loki ‘my stuff’ once you have spent over a millennia together and you never stop loving him even when he stabs you in the back figuratively and literally. You’ll need to sacrifice everything to balance protecting the realms from Loki and protecting Loki from himself. Loki requires commitment. You gotta earn the right to call Loki your stuff, y’know.