G-man is ruthless. It's all about the bottom line which is performing.
As for the office, I think that he is even more attuned to what's going on, and he is not going to be as quick to ax someone if he is the one that's ultimately making the call. Unlike the field which the world can see, we don't exactly know what's going on in the office. It could be that Rivera and other coaches just can't do miracles with the talent that G-man was adamant about getting. In that case, how can he expect for someone else to be the fall guy? Moreover, when you lose basically three Pro-Bowl level talents like Gross, Hardy, and Smitty (yes, Smitty), any fool should look for some type of decline.
Bottom line: If someone just basically is under performing with the tools and or talent that they've been given, I think that G-man will fire them without much mercy. It's a business.