It's fairly easy to see that the show has become it's own animal. It took it a few seasons to get there, and I'm actually a little surprised with some of the stuff that has been done so far this season. Season one was an almost word for word and scene for scene re-enactment of the first book. It was good, and fun to see the book come to life, but it wasn't exciting for anyone that had read the book, IMHO. I'm kind of glad that they are going off course a little. It will keep it fresh, and makes it fun to watch.
As long as they don't give it the Walking Dead treatment and create totally new main characters, and as long as they hit the main points (which so far they are), there's no big deal with taking some liberties and adding or deleting a scene here or there. Imagine if they had followed Bran's journey north by the book, that would have been incredibly snooze-worthy.